Apr  2 00:01:00 rdma06 systemd[1]: Starting Rotate log files...
Apr  2 00:01:00 rdma06 systemd[1]: rsyslog.service: Sent signal SIGHUP to main process 1041 (rsyslogd) on client request.
Apr  2 00:01:01 rdma06 systemd[1]: logrotate.service: Deactivated successfully.
Apr  2 00:01:01 rdma06 systemd[1]: Finished Rotate log files.
Apr  2 00:02:46 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 00:02:46 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 00:02:46 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 00:02:46 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 00:07:46 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 00:07:46 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 00:07:46 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 00:07:46 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 00:11:00 rdma06 rsyslogd[1041]: [origin software="rsyslogd" swVersion="8.2102.0-113.el9_2" x-pid="1041" x-info="https://www.rsyslog.com"] rsyslogd was HUPed
Apr  2 00:12:46 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 00:12:46 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 00:12:46 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 00:12:46 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 00:17:46 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 00:17:46 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 00:17:46 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 00:17:46 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 00:22:46 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 00:22:46 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 00:22:46 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 00:22:47 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 00:27:47 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 00:27:47 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 00:27:47 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 00:27:47 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 00:32:47 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 00:32:47 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 00:32:47 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 00:32:47 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 00:37:47 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 00:37:47 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 00:37:47 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 00:37:47 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 00:42:47 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 00:42:47 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 00:42:47 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 00:42:47 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 00:47:47 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 00:47:47 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 00:47:47 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 00:47:47 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 00:52:47 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 00:52:47 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 00:52:47 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 00:52:47 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 00:57:47 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 00:57:47 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 00:57:47 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 00:57:47 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 01:02:40 rdma06 systemd[1]: Starting dnf makecache...
Apr  2 01:02:41 rdma06 dnf[258237]: Updating Subscription Management repositories.
Apr  2 01:02:41 rdma06 dnf[258237]: Unable to read consumer identity
Apr  2 01:02:41 rdma06 dnf[258237]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  2 01:02:41 rdma06 dnf[258237]: Metadata cache refreshed recently.
Apr  2 01:02:41 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  2 01:02:41 rdma06 systemd[1]: Finished dnf makecache.
Apr  2 01:02:47 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 01:02:47 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 01:02:47 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 01:02:47 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 01:07:47 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 01:07:47 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 01:07:47 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 01:07:47 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 01:12:47 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 01:12:47 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 01:12:47 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 01:12:48 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 01:17:48 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 01:17:48 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 01:17:48 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 01:17:48 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 01:22:48 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 01:22:48 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 01:22:48 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 01:22:48 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 01:27:48 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 01:27:48 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 01:27:48 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 01:27:48 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 01:32:48 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 01:32:48 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 01:32:48 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 01:32:48 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 01:37:48 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 01:37:48 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 01:37:48 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 01:37:48 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 01:42:48 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 01:42:48 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 01:42:48 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 01:42:48 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 01:47:48 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 01:47:48 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 01:47:48 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 01:47:48 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 01:52:48 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 01:52:48 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 01:52:48 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 01:52:48 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 01:57:48 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 01:57:48 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 01:57:48 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 01:57:48 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 02:02:48 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 02:02:48 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 02:02:48 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 02:02:49 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 02:07:49 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 02:07:49 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 02:07:49 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 02:07:49 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 02:12:49 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 02:12:49 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 02:12:49 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 02:12:49 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 02:17:49 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 02:17:49 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 02:17:49 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 02:17:49 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 02:22:49 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 02:22:49 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 02:22:49 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 02:22:49 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 02:27:49 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 02:27:49 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 02:27:49 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 02:27:49 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 02:32:49 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 02:32:49 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 02:32:49 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 02:32:49 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 02:37:49 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 02:37:49 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 02:37:49 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 02:37:49 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 02:42:49 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 02:42:49 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 02:42:49 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 02:42:49 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 02:47:49 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 02:47:49 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 02:47:49 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 02:47:49 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 02:52:49 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 02:52:49 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 02:52:49 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 02:52:50 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 02:57:50 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 02:57:50 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 02:57:50 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 02:57:50 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 03:01:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  2 03:01:01 rdma06 dnf[263047]: Updating Subscription Management repositories.
Apr  2 03:01:01 rdma06 dnf[263047]: Unable to read consumer identity
Apr  2 03:01:01 rdma06 dnf[263047]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  2 03:01:03 rdma06 dnf[263047]: Copr repo for fmf owned by psss                 1.6 kB/s | 3.3 kB     00:02
Apr  2 03:01:03 rdma06 dnf[263047]: beaker-AppStream                                 26 kB/s | 3.2 kB     00:00
Apr  2 03:01:03 rdma06 dnf[263047]: beaker-AppStream-debuginfo                       35 kB/s | 1.5 kB     00:00
Apr  2 03:01:03 rdma06 dnf[263047]: beaker-BaseOS                                    85 kB/s | 2.7 kB     00:00
Apr  2 03:01:03 rdma06 dnf[263047]: beaker-BaseOS-debuginfo                          44 kB/s | 1.5 kB     00:00
Apr  2 03:01:04 rdma06 dnf[263047]: beaker-CRB                                       53 kB/s | 2.7 kB     00:00
Apr  2 03:01:04 rdma06 dnf[263047]: beaker-CRB-debuginfo                             63 kB/s | 1.5 kB     00:00
Apr  2 03:01:04 rdma06 dnf[263047]: beaker-HighAvailability                          59 kB/s | 2.8 kB     00:00
Apr  2 03:01:04 rdma06 dnf[263047]: beaker-HighAvailability-debuginfo                38 kB/s | 1.5 kB     00:00
Apr  2 03:01:04 rdma06 dnf[263047]: beaker-NFV                                       35 kB/s | 2.7 kB     00:00
Apr  2 03:01:04 rdma06 dnf[263047]: beaker-NFV-debuginfo                             51 kB/s | 1.5 kB     00:00
Apr  2 03:01:04 rdma06 dnf[263047]: beaker-RT                                        55 kB/s | 2.7 kB     00:00
Apr  2 03:01:04 rdma06 dnf[263047]: beaker-RT-debuginfo                              33 kB/s | 1.5 kB     00:00
Apr  2 03:01:04 rdma06 dnf[263047]: beaker-ResilientStorage                          45 kB/s | 2.8 kB     00:00
Apr  2 03:01:04 rdma06 dnf[263047]: beaker-ResilientStorage-debuginfo                42 kB/s | 1.5 kB     00:00
Apr  2 03:01:04 rdma06 dnf[263047]: beaker-SAP                                       53 kB/s | 2.7 kB     00:00
Apr  2 03:01:04 rdma06 dnf[263047]: beaker-SAP-debuginfo                             48 kB/s | 1.5 kB     00:00
Apr  2 03:01:04 rdma06 dnf[263047]: beaker-SAPHANA                                  109 kB/s | 2.7 kB     00:00
Apr  2 03:01:04 rdma06 dnf[263047]: beaker-SAPHANA-debuginfo                         40 kB/s | 1.5 kB     00:00
Apr  2 03:01:05 rdma06 dnf[263047]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr  2 03:01:06 rdma06 dnf[263047]: beaker-tasks                                    2.1 kB/s | 1.3 kB     00:00
Apr  2 03:01:06 rdma06 dnf[263047]: Metadata cache created.
Apr  2 03:01:06 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  2 03:01:06 rdma06 systemd[1]: Finished dnf makecache.
Apr  2 03:01:06 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.383s CPU time.
Apr  2 03:02:50 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 03:02:50 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 03:02:50 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 03:02:50 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 03:07:50 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 03:07:50 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 03:07:50 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 03:07:50 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 03:12:50 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 03:12:50 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 03:12:50 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 03:12:50 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 03:17:50 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 03:17:50 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 03:17:50 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 03:17:50 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 03:22:50 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 03:22:50 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 03:22:50 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 03:22:50 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 03:27:50 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 03:27:50 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 03:27:50 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 03:27:50 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 03:32:50 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 03:32:50 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 03:32:50 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 03:32:50 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 03:37:50 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 03:37:50 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 03:37:50 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 03:37:50 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 03:42:50 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 03:42:50 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 03:42:50 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 03:42:51 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 03:47:51 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 03:47:51 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 03:47:51 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 03:47:51 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 03:52:51 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 03:52:51 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 03:52:51 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 03:52:51 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 03:57:51 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 03:57:51 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 03:57:51 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 03:57:51 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 04:02:51 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 04:02:51 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 04:02:51 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 04:02:51 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 04:07:51 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 04:07:51 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 04:07:51 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 04:07:51 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 04:12:51 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 04:12:51 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 04:12:51 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 04:12:51 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 04:14:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  2 04:14:01 rdma06 dnf[266032]: Updating Subscription Management repositories.
Apr  2 04:14:01 rdma06 dnf[266032]: Unable to read consumer identity
Apr  2 04:14:01 rdma06 dnf[266032]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  2 04:14:01 rdma06 dnf[266032]: Metadata cache refreshed recently.
Apr  2 04:14:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  2 04:14:01 rdma06 systemd[1]: Finished dnf makecache.
Apr  2 04:17:51 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 04:17:51 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 04:17:51 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 04:17:51 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.056 sec sweep
Apr  2 04:20:32 rdma06 NetworkManager[1025]: <info>  [1680423632.9375] dhcp4 (eno1): state changed new lease, address=10.73.131.45
Apr  2 04:22:51 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 04:22:51 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 04:22:51 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 04:22:51 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 04:27:51 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 04:27:51 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 04:27:51 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 04:27:51 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 04:32:51 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 04:32:51 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 04:32:51 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 04:32:52 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 04:37:52 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 04:37:52 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 04:37:52 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 04:37:52 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 04:42:52 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 04:42:52 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 04:42:52 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 04:42:52 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 04:47:52 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 04:47:52 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 04:47:52 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 04:47:52 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 04:52:52 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 04:52:52 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 04:52:52 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 04:52:52 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 04:57:52 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 04:57:52 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 04:57:52 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 04:57:52 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 05:02:52 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 05:02:52 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 05:02:52 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 05:02:52 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 05:07:52 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 05:07:52 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 05:07:52 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 05:07:52 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 05:12:52 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 05:12:52 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 05:12:52 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 05:12:52 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 05:17:52 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 05:17:52 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 05:17:52 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 05:17:52 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 05:20:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  2 05:20:01 rdma06 dnf[268694]: Updating Subscription Management repositories.
Apr  2 05:20:01 rdma06 dnf[268694]: Unable to read consumer identity
Apr  2 05:20:01 rdma06 dnf[268694]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  2 05:20:01 rdma06 dnf[268694]: Metadata cache refreshed recently.
Apr  2 05:20:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  2 05:20:01 rdma06 systemd[1]: Finished dnf makecache.
Apr  2 05:22:52 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 05:22:52 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 05:22:52 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 05:22:52 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 05:27:53 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 05:27:53 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 05:27:53 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 05:27:53 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 05:32:53 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 05:32:53 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 05:32:53 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 05:32:53 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 05:37:53 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 05:37:53 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 05:37:53 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 05:37:53 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 05:42:53 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 05:42:53 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 05:42:53 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 05:42:53 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 05:47:53 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 05:47:53 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 05:47:53 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 05:47:53 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 05:52:53 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 05:52:53 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 05:52:53 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 05:52:53 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 05:57:53 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 05:57:53 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 05:57:53 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 05:57:53 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 06:02:53 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 06:02:53 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 06:02:53 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 06:02:53 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 06:07:53 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 06:07:53 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 06:07:53 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 06:07:53 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 06:12:53 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 06:12:53 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 06:12:53 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 06:12:53 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 06:17:54 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 06:17:54 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 06:17:54 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 06:17:54 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 06:22:54 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 06:22:54 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 06:22:54 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 06:22:54 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 06:27:54 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 06:27:54 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 06:27:54 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 06:27:54 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 06:32:54 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 06:32:54 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 06:32:54 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 06:32:54 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 06:37:54 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 06:37:54 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 06:37:54 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 06:37:54 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 06:42:54 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 06:42:54 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 06:42:54 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 06:42:54 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 06:47:54 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 06:47:54 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 06:47:54 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 06:47:54 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 06:52:54 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 06:52:54 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 06:52:54 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 06:52:54 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 06:53:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  2 06:53:01 rdma06 dnf[272441]: Updating Subscription Management repositories.
Apr  2 06:53:01 rdma06 dnf[272441]: Unable to read consumer identity
Apr  2 06:53:01 rdma06 dnf[272441]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  2 06:53:02 rdma06 dnf[272441]: Copr repo for fmf owned by psss                 2.7 kB/s | 3.3 kB     00:01
Apr  2 06:53:02 rdma06 dnf[272441]: beaker-AppStream                                 35 kB/s | 3.2 kB     00:00
Apr  2 06:53:03 rdma06 dnf[272441]: beaker-AppStream-debuginfo                       35 kB/s | 1.5 kB     00:00
Apr  2 06:53:03 rdma06 dnf[272441]: beaker-BaseOS                                    82 kB/s | 2.7 kB     00:00
Apr  2 06:53:03 rdma06 dnf[272441]: beaker-BaseOS-debuginfo                          40 kB/s | 1.5 kB     00:00
Apr  2 06:53:03 rdma06 dnf[272441]: beaker-CRB                                       40 kB/s | 2.7 kB     00:00
Apr  2 06:53:03 rdma06 dnf[272441]: beaker-CRB-debuginfo                             72 kB/s | 1.5 kB     00:00
Apr  2 06:53:03 rdma06 dnf[272441]: beaker-HighAvailability                          71 kB/s | 2.8 kB     00:00
Apr  2 06:53:03 rdma06 dnf[272441]: beaker-HighAvailability-debuginfo                51 kB/s | 1.5 kB     00:00
Apr  2 06:53:03 rdma06 dnf[272441]: beaker-NFV                                       75 kB/s | 2.7 kB     00:00
Apr  2 06:53:03 rdma06 dnf[272441]: beaker-NFV-debuginfo                             77 kB/s | 1.5 kB     00:00
Apr  2 06:53:03 rdma06 dnf[272441]: beaker-RT                                        73 kB/s | 2.7 kB     00:00
Apr  2 06:53:03 rdma06 dnf[272441]: beaker-RT-debuginfo                              34 kB/s | 1.5 kB     00:00
Apr  2 06:53:03 rdma06 dnf[272441]: beaker-ResilientStorage                          64 kB/s | 2.8 kB     00:00
Apr  2 06:53:03 rdma06 dnf[272441]: beaker-ResilientStorage-debuginfo                55 kB/s | 1.5 kB     00:00
Apr  2 06:53:03 rdma06 dnf[272441]: beaker-SAP                                       77 kB/s | 2.7 kB     00:00
Apr  2 06:53:03 rdma06 dnf[272441]: beaker-SAP-debuginfo                             60 kB/s | 1.5 kB     00:00
Apr  2 06:53:03 rdma06 dnf[272441]: beaker-SAPHANA                                   85 kB/s | 2.7 kB     00:00
Apr  2 06:53:03 rdma06 dnf[272441]: beaker-SAPHANA-debuginfo                         39 kB/s | 1.5 kB     00:00
Apr  2 06:53:04 rdma06 dnf[272441]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr  2 06:53:05 rdma06 dnf[272441]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr  2 06:53:05 rdma06 dnf[272441]: Metadata cache created.
Apr  2 06:53:05 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  2 06:53:05 rdma06 systemd[1]: Finished dnf makecache.
Apr  2 06:53:05 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.464s CPU time.
Apr  2 06:57:54 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 06:57:54 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 06:57:54 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 06:57:54 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.056 sec sweep
Apr  2 07:02:54 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 07:02:54 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 07:02:54 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 07:02:54 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 07:07:54 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 07:07:54 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 07:07:54 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 07:07:55 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 07:12:55 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 07:12:55 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 07:12:55 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 07:12:55 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 07:17:55 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 07:17:55 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 07:17:55 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 07:17:55 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 07:22:55 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 07:22:55 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 07:22:55 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 07:22:55 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 07:27:55 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 07:27:55 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 07:27:55 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 07:27:55 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 07:32:55 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 07:32:55 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 07:32:55 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 07:32:55 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 07:37:55 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 07:37:55 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 07:37:55 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 07:37:55 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 07:42:55 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 07:42:55 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 07:42:55 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 07:42:55 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 07:47:55 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 07:47:55 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 07:47:55 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 07:47:55 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 07:52:55 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 07:52:55 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 07:52:55 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 07:52:55 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 07:57:55 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 07:57:55 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 07:57:55 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 07:57:55 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 08:02:56 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 08:02:56 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 08:02:56 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 08:02:56 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 08:07:56 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 08:07:56 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 08:07:56 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 08:07:56 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 08:12:56 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 08:12:56 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 08:12:56 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 08:12:56 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 08:17:56 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 08:17:56 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 08:17:56 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 08:17:56 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 08:22:56 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 08:22:56 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 08:22:56 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 08:22:56 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 08:27:56 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 08:27:56 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 08:27:56 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 08:27:56 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 08:32:56 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 08:32:56 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 08:32:56 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 08:32:56 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 08:37:56 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 08:37:56 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 08:37:56 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 08:37:56 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 08:42:56 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 08:42:56 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 08:42:56 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 08:42:56 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 08:46:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  2 08:46:01 rdma06 dnf[277017]: Updating Subscription Management repositories.
Apr  2 08:46:01 rdma06 dnf[277017]: Unable to read consumer identity
Apr  2 08:46:01 rdma06 dnf[277017]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  2 08:46:01 rdma06 dnf[277017]: Metadata cache refreshed recently.
Apr  2 08:46:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  2 08:46:01 rdma06 systemd[1]: Finished dnf makecache.
Apr  2 08:47:56 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 08:47:56 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 08:47:56 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 08:47:56 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.056 sec sweep
Apr  2 08:52:57 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 08:52:57 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 08:52:57 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 08:52:57 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 08:57:57 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 08:57:57 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 08:57:57 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 08:57:57 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 09:02:57 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 09:02:57 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 09:02:57 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 09:02:57 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 09:07:57 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 09:07:57 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 09:07:57 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 09:07:57 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.056 sec sweep
Apr  2 09:12:57 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 09:12:57 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 09:12:57 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 09:12:57 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 09:17:57 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 09:17:57 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 09:17:57 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 09:17:57 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 09:22:57 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 09:22:57 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 09:22:57 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 09:22:57 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 09:27:57 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 09:27:57 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 09:27:57 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 09:27:57 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 09:32:57 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 09:32:57 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 09:32:57 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 09:32:57 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 09:37:57 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 09:37:57 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 09:37:57 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 09:37:57 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 09:42:57 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 09:42:57 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 09:42:57 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 09:42:58 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 09:47:58 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 09:47:58 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 09:47:58 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 09:47:58 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 09:52:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  2 09:52:01 rdma06 dnf[279678]: Updating Subscription Management repositories.
Apr  2 09:52:01 rdma06 dnf[279678]: Unable to read consumer identity
Apr  2 09:52:01 rdma06 dnf[279678]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  2 09:52:01 rdma06 dnf[279678]: Metadata cache refreshed recently.
Apr  2 09:52:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  2 09:52:01 rdma06 systemd[1]: Finished dnf makecache.
Apr  2 09:52:58 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 09:52:58 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 09:52:58 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 09:52:58 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 09:57:58 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 09:57:58 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 09:57:58 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 09:57:58 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 10:02:58 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 10:02:58 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 10:02:58 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 10:02:58 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 10:07:58 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 10:07:58 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 10:07:58 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 10:07:58 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 10:12:58 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 10:12:58 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 10:12:58 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 10:12:58 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 10:17:58 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 10:17:58 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 10:17:58 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 10:17:58 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 10:20:32 rdma06 NetworkManager[1025]: <info>  [1680445232.9456] dhcp4 (eno1): state changed new lease, address=10.73.131.45
Apr  2 10:22:58 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 10:22:58 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 10:22:58 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 10:22:58 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 10:27:58 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 10:27:58 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 10:27:58 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 10:27:58 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 10:32:58 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 10:32:58 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 10:32:58 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 10:32:58 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 10:37:59 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 10:37:59 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 10:37:59 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 10:37:59 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 10:42:59 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 10:42:59 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 10:42:59 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 10:42:59 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 10:47:59 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 10:47:59 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 10:47:59 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 10:47:59 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 10:52:59 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 10:52:59 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 10:52:59 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 10:52:59 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 10:57:59 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 10:57:59 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 10:57:59 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 10:57:59 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 11:02:59 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 11:02:59 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 11:02:59 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 11:02:59 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 11:07:59 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 11:07:59 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 11:07:59 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 11:07:59 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 11:12:59 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 11:12:59 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 11:12:59 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 11:12:59 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 11:17:59 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 11:17:59 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 11:17:59 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 11:17:59 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 11:22:59 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 11:22:59 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 11:22:59 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 11:22:59 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 11:27:59 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 11:27:59 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 11:27:59 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 11:28:00 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 11:33:00 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 11:33:00 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 11:33:00 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 11:33:00 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 11:38:00 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 11:38:00 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 11:38:00 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 11:38:00 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 11:43:00 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 11:43:00 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 11:43:00 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 11:43:00 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 11:48:00 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 11:48:00 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 11:48:00 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 11:48:00 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 11:52:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  2 11:52:01 rdma06 dnf[284518]: Updating Subscription Management repositories.
Apr  2 11:52:01 rdma06 dnf[284518]: Unable to read consumer identity
Apr  2 11:52:01 rdma06 dnf[284518]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  2 11:52:02 rdma06 dnf[284518]: Copr repo for fmf owned by psss                 3.0 kB/s | 3.3 kB     00:01
Apr  2 11:52:02 rdma06 dnf[284518]: beaker-AppStream                                 55 kB/s | 3.2 kB     00:00
Apr  2 11:52:02 rdma06 dnf[284518]: beaker-AppStream-debuginfo                       57 kB/s | 1.5 kB     00:00
Apr  2 11:52:02 rdma06 dnf[284518]: beaker-BaseOS                                   150 kB/s | 2.7 kB     00:00
Apr  2 11:52:02 rdma06 dnf[284518]: beaker-BaseOS-debuginfo                          34 kB/s | 1.5 kB     00:00
Apr  2 11:52:02 rdma06 dnf[284518]: beaker-CRB                                       56 kB/s | 2.7 kB     00:00
Apr  2 11:52:03 rdma06 dnf[284518]: beaker-CRB-debuginfo                             79 kB/s | 1.5 kB     00:00
Apr  2 11:52:03 rdma06 dnf[284518]: beaker-HighAvailability                          67 kB/s | 2.8 kB     00:00
Apr  2 11:52:03 rdma06 dnf[284518]: beaker-HighAvailability-debuginfo                48 kB/s | 1.5 kB     00:00
Apr  2 11:52:03 rdma06 dnf[284518]: beaker-NFV                                       66 kB/s | 2.7 kB     00:00
Apr  2 11:52:03 rdma06 dnf[284518]: beaker-NFV-debuginfo                             58 kB/s | 1.5 kB     00:00
Apr  2 11:52:03 rdma06 dnf[284518]: beaker-RT                                        55 kB/s | 2.7 kB     00:00
Apr  2 11:52:03 rdma06 dnf[284518]: beaker-RT-debuginfo                              37 kB/s | 1.5 kB     00:00
Apr  2 11:52:03 rdma06 dnf[284518]: beaker-ResilientStorage                          63 kB/s | 2.8 kB     00:00
Apr  2 11:52:03 rdma06 dnf[284518]: beaker-ResilientStorage-debuginfo                54 kB/s | 1.5 kB     00:00
Apr  2 11:52:03 rdma06 dnf[284518]: beaker-SAP                                      149 kB/s | 2.7 kB     00:00
Apr  2 11:52:03 rdma06 dnf[284518]: beaker-SAP-debuginfo                             59 kB/s | 1.5 kB     00:00
Apr  2 11:52:03 rdma06 dnf[284518]: beaker-SAPHANA                                   73 kB/s | 2.7 kB     00:00
Apr  2 11:52:03 rdma06 dnf[284518]: beaker-SAPHANA-debuginfo                         43 kB/s | 1.5 kB     00:00
Apr  2 11:52:04 rdma06 dnf[284518]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr  2 11:52:04 rdma06 dnf[284518]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr  2 11:52:05 rdma06 dnf[284518]: Metadata cache created.
Apr  2 11:52:05 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  2 11:52:05 rdma06 systemd[1]: Finished dnf makecache.
Apr  2 11:52:05 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.344s CPU time.
Apr  2 11:53:00 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 11:53:00 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 11:53:00 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 11:53:00 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.056 sec sweep
Apr  2 11:58:00 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 11:58:00 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 11:58:00 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 11:58:00 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 12:03:00 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 12:03:00 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 12:03:00 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 12:03:00 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 12:08:00 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 12:08:00 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 12:08:00 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 12:08:00 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 12:13:00 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 12:13:00 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 12:13:00 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 12:13:00 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 12:18:00 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 12:18:00 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 12:18:00 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 12:18:01 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 12:23:01 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 12:23:01 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 12:23:01 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 12:23:01 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 12:28:01 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 12:28:01 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 12:28:01 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 12:28:01 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 12:33:01 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 12:33:01 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 12:33:01 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 12:33:01 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 12:38:01 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 12:38:01 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 12:38:01 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 12:38:01 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 12:43:01 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 12:43:01 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 12:43:01 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 12:43:01 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 12:48:01 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 12:48:01 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 12:48:01 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 12:48:01 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 12:53:01 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 12:53:01 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 12:53:01 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 12:53:01 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 12:58:01 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 12:58:01 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 12:58:01 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 12:58:01 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 13:03:01 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 13:03:01 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 13:03:01 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 13:03:01 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 13:08:01 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 13:08:01 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 13:08:01 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 13:08:01 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 13:13:02 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 13:13:02 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 13:13:02 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 13:13:02 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 13:18:02 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 13:18:02 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 13:18:02 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 13:18:02 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 13:23:02 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 13:23:02 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 13:23:02 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 13:23:02 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 13:26:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  2 13:26:01 rdma06 dnf[288339]: Updating Subscription Management repositories.
Apr  2 13:26:01 rdma06 dnf[288339]: Unable to read consumer identity
Apr  2 13:26:01 rdma06 dnf[288339]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  2 13:26:01 rdma06 dnf[288339]: Metadata cache refreshed recently.
Apr  2 13:26:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  2 13:26:01 rdma06 systemd[1]: Finished dnf makecache.
Apr  2 13:28:02 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 13:28:02 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 13:28:02 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 13:28:02 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.056 sec sweep
Apr  2 13:33:02 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 13:33:02 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 13:33:02 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 13:33:02 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 13:38:02 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 13:38:02 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 13:38:02 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 13:38:02 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 13:43:02 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 13:43:02 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 13:43:02 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 13:43:02 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 13:48:02 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 13:48:02 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 13:48:02 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 13:48:02 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 13:53:02 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 13:53:02 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 13:53:02 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 13:53:02 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 13:58:02 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 13:58:02 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 13:58:02 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 13:58:02 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 14:03:03 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 14:03:03 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 14:03:03 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 14:03:03 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 14:08:03 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 14:08:03 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 14:08:03 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 14:08:03 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 14:13:03 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 14:13:03 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 14:13:03 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 14:13:03 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 14:18:03 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 14:18:03 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 14:18:03 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 14:18:03 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 14:23:03 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 14:23:03 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 14:23:03 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 14:23:03 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 14:28:03 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 14:28:03 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 14:28:03 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 14:28:03 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 14:33:03 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 14:33:03 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 14:33:03 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 14:33:03 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 14:38:03 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 14:38:03 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 14:38:03 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 14:38:03 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 14:43:03 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 14:43:03 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 14:43:03 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 14:43:03 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 14:45:47 rdma06 chronyd[1034]: Selected source 10.11.160.238
Apr  2 14:48:03 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 14:48:03 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 14:48:03 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 14:48:03 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 14:53:04 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 14:53:04 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 14:53:04 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 14:53:04 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 14:58:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  2 14:58:01 rdma06 dnf[292044]: Updating Subscription Management repositories.
Apr  2 14:58:01 rdma06 dnf[292044]: Unable to read consumer identity
Apr  2 14:58:01 rdma06 dnf[292044]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  2 14:58:03 rdma06 dnf[292044]: Copr repo for fmf owned by psss                 2.1 kB/s | 3.3 kB     00:01
Apr  2 14:58:03 rdma06 dnf[292044]: beaker-AppStream                                 33 kB/s | 3.2 kB     00:00
Apr  2 14:58:03 rdma06 dnf[292044]: beaker-AppStream-debuginfo                       39 kB/s | 1.5 kB     00:00
Apr  2 14:58:03 rdma06 dnf[292044]: beaker-BaseOS                                    70 kB/s | 2.7 kB     00:00
Apr  2 14:58:03 rdma06 dnf[292044]: beaker-BaseOS-debuginfo                          43 kB/s | 1.5 kB     00:00
Apr  2 14:58:03 rdma06 dnf[292044]: beaker-CRB                                       59 kB/s | 2.7 kB     00:00
Apr  2 14:58:03 rdma06 dnf[292044]: beaker-CRB-debuginfo                             79 kB/s | 1.5 kB     00:00
Apr  2 14:58:03 rdma06 dnf[292044]: beaker-HighAvailability                          71 kB/s | 2.8 kB     00:00
Apr  2 14:58:03 rdma06 dnf[292044]: beaker-HighAvailability-debuginfo                48 kB/s | 1.5 kB     00:00
Apr  2 14:58:03 rdma06 dnf[292044]: beaker-NFV                                       85 kB/s | 2.7 kB     00:00
Apr  2 14:58:03 rdma06 dnf[292044]: beaker-NFV-debuginfo                             61 kB/s | 1.5 kB     00:00
Apr  2 14:58:03 rdma06 dnf[292044]: beaker-RT                                        48 kB/s | 2.7 kB     00:00
Apr  2 14:58:03 rdma06 dnf[292044]: beaker-RT-debuginfo                              67 kB/s | 1.5 kB     00:00
Apr  2 14:58:03 rdma06 dnf[292044]: beaker-ResilientStorage                          81 kB/s | 2.8 kB     00:00
Apr  2 14:58:03 rdma06 dnf[292044]: beaker-ResilientStorage-debuginfo                66 kB/s | 1.5 kB     00:00
Apr  2 14:58:03 rdma06 dnf[292044]: beaker-SAP                                      181 kB/s | 2.7 kB     00:00
Apr  2 14:58:03 rdma06 dnf[292044]: beaker-SAP-debuginfo                             60 kB/s | 1.5 kB     00:00
Apr  2 14:58:04 rdma06 dnf[292044]: beaker-SAPHANA                                   65 kB/s | 2.7 kB     00:00
Apr  2 14:58:04 rdma06 dnf[292044]: beaker-SAPHANA-debuginfo                         43 kB/s | 1.5 kB     00:00
Apr  2 14:58:04 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 14:58:04 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 14:58:04 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 14:58:04 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 14:58:04 rdma06 dnf[292044]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr  2 14:58:05 rdma06 dnf[292044]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr  2 14:58:05 rdma06 dnf[292044]: Metadata cache created.
Apr  2 14:58:05 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  2 14:58:05 rdma06 systemd[1]: Finished dnf makecache.
Apr  2 14:58:05 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.391s CPU time.
Apr  2 15:03:04 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 15:03:04 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 15:03:04 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 15:03:04 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 15:08:04 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 15:08:04 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 15:08:04 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 15:08:04 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 15:13:04 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 15:13:04 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 15:13:04 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 15:13:04 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 15:18:04 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 15:18:04 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 15:18:04 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 15:18:04 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 15:23:04 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 15:23:04 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 15:23:04 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 15:23:04 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 15:28:04 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 15:28:04 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 15:28:04 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 15:28:04 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 15:33:04 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 15:33:04 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 15:33:04 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 15:33:04 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 15:38:04 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 15:38:04 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 15:38:04 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 15:38:04 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 15:43:04 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 15:43:04 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 15:43:04 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 15:43:05 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 15:48:05 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 15:48:05 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 15:48:05 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 15:48:05 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 15:53:05 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 15:53:05 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 15:53:05 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 15:53:05 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 15:58:05 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 15:58:05 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 15:58:05 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 15:58:05 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 16:03:05 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 16:03:05 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 16:03:05 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 16:03:05 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 16:08:05 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 16:08:05 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 16:08:05 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 16:08:05 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 16:13:05 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 16:13:05 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 16:13:05 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 16:13:05 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 16:18:05 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 16:18:05 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 16:18:05 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 16:18:05 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 16:20:32 rdma06 NetworkManager[1025]: <info>  [1680466832.9509] dhcp4 (eno1): state changed new lease, address=10.73.131.45
Apr  2 16:23:05 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 16:23:05 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 16:23:05 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 16:23:05 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 16:28:05 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 16:28:05 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 16:28:05 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 16:28:05 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 16:29:50 rdma06 systemd[1]: Starting dnf makecache...
Apr  2 16:29:51 rdma06 dnf[295766]: Updating Subscription Management repositories.
Apr  2 16:29:51 rdma06 dnf[295766]: Unable to read consumer identity
Apr  2 16:29:51 rdma06 dnf[295766]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  2 16:29:51 rdma06 dnf[295766]: Metadata cache refreshed recently.
Apr  2 16:29:51 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  2 16:29:51 rdma06 systemd[1]: Finished dnf makecache.
Apr  2 16:33:05 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 16:33:05 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 16:33:05 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 16:33:05 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 16:38:05 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 16:38:05 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 16:38:05 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 16:38:06 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 16:43:06 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 16:43:06 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 16:43:06 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 16:43:06 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 16:48:06 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 16:48:06 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 16:48:06 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 16:48:06 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 16:53:06 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 16:53:06 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 16:53:06 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 16:53:06 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 16:58:06 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 16:58:06 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 16:58:06 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 16:58:06 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 17:03:06 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 17:03:06 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 17:03:06 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 17:03:06 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 17:08:06 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 17:08:06 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 17:08:06 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 17:08:06 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 17:13:06 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 17:13:06 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 17:13:06 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 17:13:06 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 17:18:06 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 17:18:06 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 17:18:06 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 17:18:06 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 17:23:06 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 17:23:06 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 17:23:06 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 17:23:06 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 17:28:06 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 17:28:06 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 17:28:06 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 17:28:06 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 17:32:50 rdma06 systemd[1]: Starting dnf makecache...
Apr  2 17:32:51 rdma06 dnf[298307]: Updating Subscription Management repositories.
Apr  2 17:32:51 rdma06 dnf[298307]: Unable to read consumer identity
Apr  2 17:32:51 rdma06 dnf[298307]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  2 17:32:51 rdma06 dnf[298307]: Metadata cache refreshed recently.
Apr  2 17:32:51 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  2 17:32:51 rdma06 systemd[1]: Finished dnf makecache.
Apr  2 17:33:06 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 17:33:06 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 17:33:06 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 17:33:07 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 17:38:07 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 17:38:07 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 17:38:07 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 17:38:07 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 17:43:07 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 17:43:07 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 17:43:07 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 17:43:07 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 17:48:07 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 17:48:07 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 17:48:07 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 17:48:07 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 17:53:07 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 17:53:07 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 17:53:07 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 17:53:07 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 17:58:07 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 17:58:07 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 17:58:07 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 17:58:07 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 18:03:07 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 18:03:07 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 18:03:07 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 18:03:07 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 18:08:07 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 18:08:07 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 18:08:07 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 18:08:07 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 18:13:07 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 18:13:07 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 18:13:07 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 18:13:07 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 18:18:07 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 18:18:07 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 18:18:07 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 18:18:07 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 18:23:07 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 18:23:07 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 18:23:07 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 18:23:08 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 18:28:08 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 18:28:08 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 18:28:08 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 18:28:08 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 18:33:08 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 18:33:08 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 18:33:08 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 18:33:08 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 18:38:08 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 18:38:08 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 18:38:08 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 18:38:08 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 18:43:08 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 18:43:08 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 18:43:08 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 18:43:08 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 18:48:08 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 18:48:08 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 18:48:08 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 18:48:08 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 18:53:08 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 18:53:08 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 18:53:08 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 18:53:08 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 18:58:08 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 18:58:08 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 18:58:08 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 18:58:08 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 19:02:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  2 19:02:01 rdma06 dnf[301913]: Updating Subscription Management repositories.
Apr  2 19:02:01 rdma06 dnf[301913]: Unable to read consumer identity
Apr  2 19:02:01 rdma06 dnf[301913]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  2 19:02:02 rdma06 dnf[301913]: Copr repo for fmf owned by psss                 6.3 kB/s | 3.3 kB     00:00
Apr  2 19:02:02 rdma06 dnf[301913]: beaker-AppStream                                 25 kB/s | 3.2 kB     00:00
Apr  2 19:02:02 rdma06 dnf[301913]: beaker-AppStream-debuginfo                       36 kB/s | 1.5 kB     00:00
Apr  2 19:02:02 rdma06 dnf[301913]: beaker-BaseOS                                   112 kB/s | 2.7 kB     00:00
Apr  2 19:02:02 rdma06 dnf[301913]: beaker-BaseOS-debuginfo                          38 kB/s | 1.5 kB     00:00
Apr  2 19:02:02 rdma06 dnf[301913]: beaker-CRB                                       54 kB/s | 2.7 kB     00:00
Apr  2 19:02:02 rdma06 dnf[301913]: beaker-CRB-debuginfo                             63 kB/s | 1.5 kB     00:00
Apr  2 19:02:02 rdma06 dnf[301913]: beaker-HighAvailability                          72 kB/s | 2.8 kB     00:00
Apr  2 19:02:02 rdma06 dnf[301913]: beaker-HighAvailability-debuginfo                41 kB/s | 1.5 kB     00:00
Apr  2 19:02:02 rdma06 dnf[301913]: beaker-NFV                                       59 kB/s | 2.7 kB     00:00
Apr  2 19:02:02 rdma06 dnf[301913]: beaker-NFV-debuginfo                             49 kB/s | 1.5 kB     00:00
Apr  2 19:02:02 rdma06 dnf[301913]: beaker-RT                                        46 kB/s | 2.7 kB     00:00
Apr  2 19:02:02 rdma06 dnf[301913]: beaker-RT-debuginfo                              25 kB/s | 1.5 kB     00:00
Apr  2 19:02:03 rdma06 dnf[301913]: beaker-ResilientStorage                          60 kB/s | 2.8 kB     00:00
Apr  2 19:02:03 rdma06 dnf[301913]: beaker-ResilientStorage-debuginfo                52 kB/s | 1.5 kB     00:00
Apr  2 19:02:03 rdma06 dnf[301913]: beaker-SAP                                      126 kB/s | 2.7 kB     00:00
Apr  2 19:02:03 rdma06 dnf[301913]: beaker-SAP-debuginfo                             59 kB/s | 1.5 kB     00:00
Apr  2 19:02:03 rdma06 dnf[301913]: beaker-SAPHANA                                   60 kB/s | 2.7 kB     00:00
Apr  2 19:02:03 rdma06 dnf[301913]: beaker-SAPHANA-debuginfo                         35 kB/s | 1.5 kB     00:00
Apr  2 19:02:03 rdma06 dnf[301913]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr  2 19:02:04 rdma06 dnf[301913]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr  2 19:02:04 rdma06 dnf[301913]: Metadata cache created.
Apr  2 19:02:04 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  2 19:02:04 rdma06 systemd[1]: Finished dnf makecache.
Apr  2 19:02:04 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.373s CPU time.
Apr  2 19:03:08 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 19:03:08 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 19:03:08 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 19:03:08 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 19:08:08 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 19:08:08 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 19:08:08 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 19:08:08 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 19:13:08 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 19:13:08 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 19:13:08 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 19:13:08 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 19:18:09 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 19:18:09 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 19:18:09 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 19:18:09 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 19:23:09 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 19:23:09 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 19:23:09 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 19:23:09 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 19:28:09 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 19:28:09 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 19:28:09 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 19:28:09 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 19:33:09 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 19:33:09 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 19:33:09 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 19:33:09 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 19:38:09 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 19:38:09 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 19:38:09 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 19:38:09 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 19:43:09 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 19:43:09 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 19:43:09 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 19:43:09 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 19:48:09 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 19:48:09 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 19:48:09 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 19:48:09 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 19:53:09 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 19:53:09 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 19:53:09 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 19:53:09 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 19:58:09 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 19:58:09 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 19:58:09 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 19:58:09 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 20:03:09 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 20:03:09 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 20:03:09 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 20:03:09 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 20:08:10 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 20:08:10 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 20:08:10 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 20:08:10 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.053 sec sweep
Apr  2 20:13:10 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 20:13:10 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 20:13:10 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 20:13:10 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 20:18:10 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 20:18:10 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 20:18:10 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 20:18:10 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 20:23:10 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 20:23:10 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 20:23:10 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 20:23:10 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 20:28:10 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 20:28:10 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 20:28:10 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 20:28:10 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 20:33:10 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 20:33:10 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 20:33:10 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 20:33:10 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.053 sec sweep
Apr  2 20:38:10 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 20:38:10 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 20:38:10 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 20:38:10 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 20:43:10 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 20:43:10 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 20:43:10 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 20:43:10 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 20:46:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  2 20:46:01 rdma06 dnf[306122]: Updating Subscription Management repositories.
Apr  2 20:46:01 rdma06 dnf[306122]: Unable to read consumer identity
Apr  2 20:46:01 rdma06 dnf[306122]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  2 20:46:01 rdma06 dnf[306122]: Metadata cache refreshed recently.
Apr  2 20:46:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  2 20:46:01 rdma06 systemd[1]: Finished dnf makecache.
Apr  2 20:48:10 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 20:48:10 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 20:48:10 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 20:48:10 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 20:53:10 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 20:53:10 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 20:53:10 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 20:53:10 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 20:58:10 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 20:58:10 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 20:58:10 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 20:58:10 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 21:03:11 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 21:03:11 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 21:03:11 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 21:03:11 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 21:08:11 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 21:08:11 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 21:08:11 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 21:08:11 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 21:13:11 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 21:13:11 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 21:13:11 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 21:13:11 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 21:18:11 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 21:18:11 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 21:18:11 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 21:18:11 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 21:23:11 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 21:23:11 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 21:23:11 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 21:23:11 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 21:28:11 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 21:28:11 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 21:28:11 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 21:28:11 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 21:33:11 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 21:33:11 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 21:33:11 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 21:33:11 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 21:38:11 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 21:38:11 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 21:38:11 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 21:38:11 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 21:43:11 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 21:43:11 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 21:43:11 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 21:43:11 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 21:48:11 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 21:48:11 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 21:48:11 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 21:48:11 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 21:53:12 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 21:53:12 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 21:53:12 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 21:53:12 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 21:58:12 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 21:58:12 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 21:58:12 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 21:58:12 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 22:03:12 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 22:03:12 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 22:03:12 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 22:03:12 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 22:07:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  2 22:07:01 rdma06 dnf[309393]: Updating Subscription Management repositories.
Apr  2 22:07:01 rdma06 dnf[309393]: Unable to read consumer identity
Apr  2 22:07:01 rdma06 dnf[309393]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  2 22:07:02 rdma06 dnf[309393]: Copr repo for fmf owned by psss                 3.9 kB/s | 3.3 kB     00:00
Apr  2 22:07:02 rdma06 dnf[309393]: beaker-AppStream                                 12 kB/s | 3.2 kB     00:00
Apr  2 22:07:02 rdma06 dnf[309393]: beaker-AppStream-debuginfo                       37 kB/s | 1.5 kB     00:00
Apr  2 22:07:02 rdma06 dnf[309393]: beaker-BaseOS                                    51 kB/s | 2.7 kB     00:00
Apr  2 22:07:04 rdma06 dnf[309393]: beaker-BaseOS-debuginfo                         1.3 kB/s | 1.5 kB     00:01
Apr  2 22:07:04 rdma06 dnf[309393]: beaker-CRB                                       20 kB/s | 2.7 kB     00:00
Apr  2 22:07:05 rdma06 dnf[309393]: beaker-CRB-debuginfo                            1.3 kB/s | 1.5 kB     00:01
Apr  2 22:07:05 rdma06 dnf[309393]: beaker-HighAvailability                         6.3 kB/s | 2.8 kB     00:00
Apr  2 22:07:06 rdma06 dnf[309393]: beaker-HighAvailability-debuginfo                32 kB/s | 1.5 kB     00:00
Apr  2 22:07:06 rdma06 dnf[309393]: beaker-NFV                                       21 kB/s | 2.7 kB     00:00
Apr  2 22:07:06 rdma06 dnf[309393]: beaker-NFV-debuginfo                            3.2 kB/s | 1.5 kB     00:00
Apr  2 22:07:08 rdma06 dnf[309393]: beaker-RT                                       1.8 kB/s | 2.7 kB     00:01
Apr  2 22:07:08 rdma06 dnf[309393]: beaker-RT-debuginfo                             4.1 kB/s | 1.5 kB     00:00
Apr  2 22:07:08 rdma06 dnf[309393]: beaker-ResilientStorage                          44 kB/s | 2.8 kB     00:00
Apr  2 22:07:08 rdma06 dnf[309393]: beaker-ResilientStorage-debuginfo               5.6 kB/s | 1.5 kB     00:00
Apr  2 22:07:08 rdma06 dnf[309393]: beaker-SAP                                       53 kB/s | 2.7 kB     00:00
Apr  2 22:07:09 rdma06 dnf[309393]: beaker-SAP-debuginfo                            7.5 kB/s | 1.5 kB     00:00
Apr  2 22:07:09 rdma06 dnf[309393]: beaker-SAPHANA                                  8.0 kB/s | 2.7 kB     00:00
Apr  2 22:07:09 rdma06 dnf[309393]: beaker-SAPHANA-debuginfo                        9.4 kB/s | 1.5 kB     00:00
Apr  2 22:07:10 rdma06 dnf[309393]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr  2 22:07:10 rdma06 dnf[309393]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr  2 22:07:11 rdma06 dnf[309393]: Metadata cache created.
Apr  2 22:07:11 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  2 22:07:11 rdma06 systemd[1]: Finished dnf makecache.
Apr  2 22:07:11 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.406s CPU time.
Apr  2 22:08:12 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 22:08:12 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 22:08:12 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 22:08:12 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 22:13:12 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 22:13:12 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 22:13:12 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 22:13:12 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 22:18:12 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 22:18:12 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 22:18:12 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 22:18:12 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 22:20:32 rdma06 NetworkManager[1025]: <info>  [1680488432.9381] dhcp4 (eno1): state changed new lease, address=10.73.131.45
Apr  2 22:23:12 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 22:23:12 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 22:23:12 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 22:23:12 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 22:28:12 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 22:28:12 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 22:28:12 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 22:28:12 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 22:31:00 rdma06 systemd[5685]: Starting Cleanup of User's Temporary Files and Directories...
Apr  2 22:31:00 rdma06 systemd[5685]: Finished Cleanup of User's Temporary Files and Directories.
Apr  2 22:33:12 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 22:33:12 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 22:33:12 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 22:33:12 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 22:38:12 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 22:38:12 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 22:38:12 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 22:38:12 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 22:38:50 rdma06 systemd[1]: Starting Cleanup of Temporary Directories...
Apr  2 22:38:50 rdma06 systemd[1]: systemd-tmpfiles-clean.service: Deactivated successfully.
Apr  2 22:38:50 rdma06 systemd[1]: Finished Cleanup of Temporary Directories.
Apr  2 22:38:50 rdma06 systemd[1]: run-credentials-systemd\x2dtmpfiles\x2dclean.service.mount: Deactivated successfully.
Apr  2 22:43:12 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 22:43:12 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 22:43:12 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 22:43:12 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 22:48:12 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 22:48:12 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 22:48:12 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 22:48:12 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 22:53:12 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 22:53:12 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 22:53:12 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 22:53:12 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 22:58:12 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 22:58:12 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 22:58:12 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 22:58:12 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 23:03:12 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 23:03:12 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 23:03:12 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 23:03:12 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 23:08:12 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 23:08:12 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 23:08:12 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 23:08:13 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 23:13:13 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 23:13:13 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 23:13:13 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 23:13:13 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 23:18:13 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 23:18:13 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 23:18:13 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 23:18:13 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 23:23:13 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 23:23:13 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 23:23:13 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 23:23:13 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 23:28:13 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 23:28:13 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 23:28:13 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 23:28:13 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 23:30:33 rdma06 sshd[312786]: main: sshd: ssh-rsa algorithm is disabled
Apr  2 23:30:37 rdma06 systemd-logind[1064]: New session 8 of user root.
Apr  2 23:30:37 rdma06 systemd[1]: Started Session 8 of User root.
Apr  2 23:30:38 rdma06 systemd[1]: Starting Hostname Service...
Apr  2 23:30:38 rdma06 systemd[1]: Started Hostname Service.
Apr  2 23:30:40 rdma06 journal[2657]: Adjustment to local watchdog ignored since 'no_localwatchdog' metadata is set
Apr  2 23:31:08 rdma06 systemd[1]: systemd-hostnamed.service: Deactivated successfully.
Apr  2 23:33:13 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 23:33:13 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 23:33:13 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 23:33:13 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.056 sec sweep
Apr  2 23:38:13 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 23:38:13 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 23:38:13 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 23:38:13 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 23:43:13 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 23:43:13 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 23:43:13 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 23:43:13 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 23:48:13 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 23:48:13 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 23:48:13 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 23:48:13 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 23:53:13 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 23:53:13 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 23:53:13 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 23:53:13 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  2 23:58:13 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  2 23:58:13 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  2 23:58:13 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  2 23:58:13 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  2 23:59:50 rdma06 systemd[1]: Starting dnf makecache...
Apr  2 23:59:51 rdma06 dnf[314030]: Updating Subscription Management repositories.
Apr  2 23:59:51 rdma06 dnf[314030]: Unable to read consumer identity
Apr  2 23:59:51 rdma06 dnf[314030]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  2 23:59:51 rdma06 dnf[314030]: Metadata cache refreshed recently.
Apr  2 23:59:51 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  2 23:59:51 rdma06 systemd[1]: Finished dnf makecache.
Apr  3 00:01:00 rdma06 systemd[1]: Starting Rotate log files...
Apr  3 00:01:00 rdma06 systemd[1]: logrotate.service: Deactivated successfully.
Apr  3 00:01:00 rdma06 systemd[1]: Finished Rotate log files.
Apr  3 00:03:13 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 00:03:13 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 00:03:13 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 00:03:13 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 00:08:14 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 00:08:14 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 00:08:14 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 00:08:14 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 00:13:14 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 00:13:14 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 00:13:14 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 00:13:14 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 00:18:14 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 00:18:14 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 00:18:14 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 00:18:14 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 00:23:14 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 00:23:14 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 00:23:14 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 00:23:14 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 00:28:14 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 00:28:14 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 00:28:14 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 00:28:14 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 00:33:14 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 00:33:14 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 00:33:14 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 00:33:14 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 00:38:14 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 00:38:14 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 00:38:14 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 00:38:14 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 00:43:14 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 00:43:14 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 00:43:14 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 00:43:14 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 00:48:14 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 00:48:14 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 00:48:14 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 00:48:14 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 00:53:14 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 00:53:14 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 00:53:14 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 00:53:14 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 00:58:14 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 00:58:14 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 00:58:14 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 00:58:14 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 01:03:15 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 01:03:15 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 01:03:15 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 01:03:15 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 01:04:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  3 01:04:01 rdma06 dnf[316648]: Updating Subscription Management repositories.
Apr  3 01:04:01 rdma06 dnf[316648]: Unable to read consumer identity
Apr  3 01:04:01 rdma06 dnf[316648]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  3 01:04:01 rdma06 dnf[316648]: Metadata cache refreshed recently.
Apr  3 01:04:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  3 01:04:01 rdma06 systemd[1]: Finished dnf makecache.
Apr  3 01:08:15 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 01:08:15 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 01:08:15 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 01:08:15 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 01:13:15 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 01:13:15 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 01:13:15 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 01:13:15 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 01:18:15 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 01:18:15 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 01:18:15 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 01:18:15 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 01:23:15 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 01:23:15 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 01:23:15 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 01:23:15 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 01:28:15 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 01:28:15 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 01:28:15 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 01:28:15 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 01:33:15 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 01:33:15 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 01:33:15 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 01:33:15 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 01:38:15 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 01:38:15 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 01:38:15 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 01:38:15 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 01:43:15 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 01:43:15 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 01:43:15 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 01:43:15 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 01:48:15 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 01:48:15 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 01:48:15 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 01:48:15 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 01:53:15 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 01:53:15 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 01:53:15 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 01:53:15 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 01:58:16 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 01:58:16 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 01:58:16 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 01:58:16 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 02:03:16 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 02:03:16 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 02:03:16 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 02:03:16 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 02:08:16 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 02:08:16 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 02:08:16 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 02:08:16 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 02:13:16 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 02:13:16 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 02:13:16 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 02:13:16 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 02:18:16 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 02:18:16 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 02:18:16 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 02:18:16 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 02:23:16 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 02:23:16 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 02:23:16 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 02:23:16 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 02:26:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  3 02:26:01 rdma06 dnf[319955]: Updating Subscription Management repositories.
Apr  3 02:26:01 rdma06 dnf[319955]: Unable to read consumer identity
Apr  3 02:26:01 rdma06 dnf[319955]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  3 02:26:03 rdma06 dnf[319955]: Copr repo for fmf owned by psss                 1.4 kB/s | 3.3 kB     00:02
Apr  3 02:26:03 rdma06 dnf[319955]: beaker-AppStream                                 33 kB/s | 3.2 kB     00:00
Apr  3 02:26:04 rdma06 dnf[319955]: beaker-AppStream-debuginfo                       33 kB/s | 1.5 kB     00:00
Apr  3 02:26:04 rdma06 dnf[319955]: beaker-BaseOS                                    98 kB/s | 2.7 kB     00:00
Apr  3 02:26:04 rdma06 dnf[319955]: beaker-BaseOS-debuginfo                          49 kB/s | 1.5 kB     00:00
Apr  3 02:26:04 rdma06 dnf[319955]: beaker-CRB                                       51 kB/s | 2.7 kB     00:00
Apr  3 02:26:04 rdma06 dnf[319955]: beaker-CRB-debuginfo                             73 kB/s | 1.5 kB     00:00
Apr  3 02:26:04 rdma06 dnf[319955]: beaker-HighAvailability                          62 kB/s | 2.8 kB     00:00
Apr  3 02:26:04 rdma06 dnf[319955]: beaker-HighAvailability-debuginfo                38 kB/s | 1.5 kB     00:00
Apr  3 02:26:04 rdma06 dnf[319955]: beaker-NFV                                       73 kB/s | 2.7 kB     00:00
Apr  3 02:26:04 rdma06 dnf[319955]: beaker-NFV-debuginfo                             46 kB/s | 1.5 kB     00:00
Apr  3 02:26:04 rdma06 dnf[319955]: beaker-RT                                        53 kB/s | 2.7 kB     00:00
Apr  3 02:26:04 rdma06 dnf[319955]: beaker-RT-debuginfo                              74 kB/s | 1.5 kB     00:00
Apr  3 02:26:04 rdma06 dnf[319955]: beaker-ResilientStorage                          69 kB/s | 2.8 kB     00:00
Apr  3 02:26:04 rdma06 dnf[319955]: beaker-ResilientStorage-debuginfo                66 kB/s | 1.5 kB     00:00
Apr  3 02:26:04 rdma06 dnf[319955]: beaker-SAP                                       61 kB/s | 2.7 kB     00:00
Apr  3 02:26:04 rdma06 dnf[319955]: beaker-SAP-debuginfo                             59 kB/s | 1.5 kB     00:00
Apr  3 02:26:04 rdma06 dnf[319955]: beaker-SAPHANA                                   57 kB/s | 2.7 kB     00:00
Apr  3 02:26:04 rdma06 dnf[319955]: beaker-SAPHANA-debuginfo                         34 kB/s | 1.5 kB     00:00
Apr  3 02:26:05 rdma06 dnf[319955]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr  3 02:26:06 rdma06 dnf[319955]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr  3 02:26:06 rdma06 dnf[319955]: Metadata cache created.
Apr  3 02:26:06 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  3 02:26:06 rdma06 systemd[1]: Finished dnf makecache.
Apr  3 02:26:06 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.403s CPU time.
Apr  3 02:28:16 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 02:28:16 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 02:28:16 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 02:28:16 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 02:33:16 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 02:33:16 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 02:33:16 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 02:33:16 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 02:38:16 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 02:38:16 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 02:38:16 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 02:38:16 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 02:43:16 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 02:43:16 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 02:43:16 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 02:43:16 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 02:48:16 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 02:48:16 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 02:48:16 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 02:48:16 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 02:53:16 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 02:53:16 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 02:53:16 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 02:53:16 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 02:58:16 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 02:58:16 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 02:58:16 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 02:58:16 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 03:03:16 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 03:03:16 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 03:03:16 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 03:03:16 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 03:08:16 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 03:08:16 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 03:08:16 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 03:08:17 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 03:13:17 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 03:13:17 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 03:13:17 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 03:13:17 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 03:18:17 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 03:18:17 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 03:18:17 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 03:18:17 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 03:23:17 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 03:23:17 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 03:23:17 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 03:23:17 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.056 sec sweep
Apr  3 03:28:17 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 03:28:17 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 03:28:17 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 03:28:17 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 03:33:17 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 03:33:17 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 03:33:17 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 03:33:17 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 03:38:17 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 03:38:17 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 03:38:17 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 03:38:17 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 03:43:17 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 03:43:17 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 03:43:17 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 03:43:17 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 03:48:17 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 03:48:17 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 03:48:17 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 03:48:17 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 03:53:17 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 03:53:17 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 03:53:17 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 03:53:17 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 03:58:17 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 03:58:17 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 03:58:17 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 03:58:17 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 04:03:17 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 04:03:17 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 04:03:17 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 04:03:17 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 04:08:17 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 04:08:17 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 04:08:17 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 04:08:17 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 04:13:17 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 04:13:17 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 04:13:17 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 04:13:17 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 04:18:18 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 04:18:18 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 04:18:18 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 04:18:18 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 04:19:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  3 04:19:01 rdma06 dnf[324565]: Updating Subscription Management repositories.
Apr  3 04:19:01 rdma06 dnf[324565]: Unable to read consumer identity
Apr  3 04:19:01 rdma06 dnf[324565]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  3 04:19:01 rdma06 dnf[324565]: Metadata cache refreshed recently.
Apr  3 04:19:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  3 04:19:01 rdma06 systemd[1]: Finished dnf makecache.
Apr  3 04:20:32 rdma06 NetworkManager[1025]: <info>  [1680510032.8768] dhcp4 (eno1): state changed new lease, address=10.73.131.45
Apr  3 04:23:18 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 04:23:18 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 04:23:18 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 04:23:18 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 04:28:18 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 04:28:18 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 04:28:18 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 04:28:18 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 04:33:18 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 04:33:18 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 04:33:18 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 04:33:18 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 04:38:18 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 04:38:18 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 04:38:18 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 04:38:18 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.056 sec sweep
Apr  3 04:43:18 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 04:43:18 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 04:43:18 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 04:43:18 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 04:48:18 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 04:48:18 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 04:48:18 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 04:48:18 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 04:53:18 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 04:53:18 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 04:53:18 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 04:53:18 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.056 sec sweep
Apr  3 04:58:18 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 04:58:18 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 04:58:18 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 04:58:18 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 05:03:18 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 05:03:18 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 05:03:18 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 05:03:18 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 05:08:18 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 05:08:18 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 05:08:18 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 05:08:18 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 05:13:18 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 05:13:18 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 05:13:18 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 05:13:18 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 05:18:19 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 05:18:19 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 05:18:19 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 05:18:19 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 05:23:19 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 05:23:19 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 05:23:19 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 05:23:19 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 05:28:19 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 05:28:19 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 05:28:19 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 05:28:19 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 05:33:19 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 05:33:19 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 05:33:19 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 05:33:19 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 05:38:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  3 05:38:01 rdma06 dnf[327756]: Updating Subscription Management repositories.
Apr  3 05:38:01 rdma06 dnf[327756]: Unable to read consumer identity
Apr  3 05:38:01 rdma06 dnf[327756]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  3 05:38:03 rdma06 dnf[327756]: Copr repo for fmf owned by psss                 2.0 kB/s | 3.3 kB     00:01
Apr  3 05:38:03 rdma06 dnf[327756]: beaker-AppStream                                 12 kB/s | 3.2 kB     00:00
Apr  3 05:38:03 rdma06 dnf[327756]: beaker-AppStream-debuginfo                       17 kB/s | 1.5 kB     00:00
Apr  3 05:38:03 rdma06 dnf[327756]: beaker-BaseOS                                   118 kB/s | 2.7 kB     00:00
Apr  3 05:38:03 rdma06 dnf[327756]: beaker-BaseOS-debuginfo                          36 kB/s | 1.5 kB     00:00
Apr  3 05:38:03 rdma06 dnf[327756]: beaker-CRB                                       58 kB/s | 2.7 kB     00:00
Apr  3 05:38:03 rdma06 dnf[327756]: beaker-CRB-debuginfo                             77 kB/s | 1.5 kB     00:00
Apr  3 05:38:03 rdma06 dnf[327756]: beaker-HighAvailability                          50 kB/s | 2.8 kB     00:00
Apr  3 05:38:04 rdma06 dnf[327756]: beaker-HighAvailability-debuginfo               2.9 kB/s | 1.5 kB     00:00
Apr  3 05:38:04 rdma06 dnf[327756]: beaker-NFV                                      8.2 kB/s | 2.7 kB     00:00
Apr  3 05:38:05 rdma06 dnf[327756]: beaker-NFV-debuginfo                             11 kB/s | 1.5 kB     00:00
Apr  3 05:38:05 rdma06 dnf[327756]: beaker-RT                                       8.7 kB/s | 2.7 kB     00:00
Apr  3 05:38:05 rdma06 dnf[327756]: beaker-RT-debuginfo                              37 kB/s | 1.5 kB     00:00
Apr  3 05:38:05 rdma06 dnf[327756]: beaker-ResilientStorage                          37 kB/s | 2.8 kB     00:00
Apr  3 05:38:05 rdma06 dnf[327756]: beaker-ResilientStorage-debuginfo                38 kB/s | 1.5 kB     00:00
Apr  3 05:38:05 rdma06 dnf[327756]: beaker-SAP                                      160 kB/s | 2.7 kB     00:00
Apr  3 05:38:05 rdma06 dnf[327756]: beaker-SAP-debuginfo                             59 kB/s | 1.5 kB     00:00
Apr  3 05:38:05 rdma06 dnf[327756]: beaker-SAPHANA                                   49 kB/s | 2.7 kB     00:00
Apr  3 05:38:05 rdma06 dnf[327756]: beaker-SAPHANA-debuginfo                         42 kB/s | 1.5 kB     00:00
Apr  3 05:38:06 rdma06 dnf[327756]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr  3 05:38:07 rdma06 dnf[327756]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr  3 05:38:07 rdma06 dnf[327756]: Metadata cache created.
Apr  3 05:38:07 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  3 05:38:07 rdma06 systemd[1]: Finished dnf makecache.
Apr  3 05:38:07 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.457s CPU time.
Apr  3 05:38:19 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 05:38:19 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 05:38:19 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 05:38:19 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.056 sec sweep
Apr  3 05:43:19 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 05:43:19 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 05:43:19 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 05:43:19 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 05:48:19 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 05:48:19 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 05:48:19 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 05:48:19 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 05:53:19 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 05:53:19 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 05:53:19 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 05:53:19 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 05:58:19 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 05:58:19 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 05:58:19 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 05:58:19 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 06:03:19 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 06:03:19 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 06:03:19 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 06:03:19 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 06:08:20 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 06:08:20 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 06:08:20 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 06:08:20 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 06:13:20 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 06:13:20 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 06:13:20 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 06:13:20 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 06:18:20 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 06:18:20 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 06:18:20 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 06:18:20 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 06:23:20 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 06:23:20 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 06:23:20 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 06:23:20 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 06:28:20 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 06:28:20 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 06:28:20 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 06:28:20 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 06:33:20 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 06:33:20 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 06:33:20 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 06:33:20 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 06:38:20 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 06:38:20 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 06:38:20 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 06:38:20 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 06:43:20 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 06:43:20 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 06:43:20 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 06:43:20 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 06:48:20 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 06:48:20 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 06:48:20 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 06:48:20 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 06:53:20 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 06:53:20 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 06:53:20 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 06:53:20 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 06:58:21 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 06:58:21 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 06:58:21 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 06:58:21 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 07:03:21 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 07:03:21 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 07:03:21 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 07:03:21 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 07:08:21 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 07:08:21 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 07:08:21 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 07:08:21 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 07:13:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  3 07:13:01 rdma06 dnf[331617]: Updating Subscription Management repositories.
Apr  3 07:13:01 rdma06 dnf[331617]: Unable to read consumer identity
Apr  3 07:13:01 rdma06 dnf[331617]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  3 07:13:01 rdma06 dnf[331617]: Metadata cache refreshed recently.
Apr  3 07:13:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  3 07:13:01 rdma06 systemd[1]: Finished dnf makecache.
Apr  3 07:13:21 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 07:13:21 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 07:13:21 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 07:13:21 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 07:18:21 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 07:18:21 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 07:18:21 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 07:18:21 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 07:23:21 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 07:23:21 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 07:23:21 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 07:23:21 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 07:28:21 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 07:28:21 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 07:28:21 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 07:28:21 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 07:33:21 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 07:33:21 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 07:33:21 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 07:33:21 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 07:38:21 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 07:38:21 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 07:38:21 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 07:38:21 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 07:43:21 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 07:43:21 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 07:43:21 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 07:43:21 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 07:48:21 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 07:48:21 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 07:48:21 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 07:48:21 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 07:53:21 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 07:53:21 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 07:53:21 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 07:53:22 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 07:58:22 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 07:58:22 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 07:58:22 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 07:58:22 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 08:03:22 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 08:03:22 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 08:03:22 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 08:03:22 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 08:08:22 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 08:08:22 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 08:08:22 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 08:08:22 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 08:13:22 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 08:13:22 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 08:13:22 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 08:13:22 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 08:18:22 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 08:18:22 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 08:18:22 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 08:18:22 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 08:23:22 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 08:23:22 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 08:23:22 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 08:23:22 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 08:28:22 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 08:28:22 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 08:28:22 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 08:28:22 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 08:33:22 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 08:33:22 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 08:33:22 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 08:33:22 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 08:38:22 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 08:38:22 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 08:38:22 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 08:38:22 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 08:39:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  3 08:39:01 rdma06 dnf[335085]: Updating Subscription Management repositories.
Apr  3 08:39:01 rdma06 dnf[335085]: Unable to read consumer identity
Apr  3 08:39:01 rdma06 dnf[335085]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  3 08:39:02 rdma06 dnf[335085]: Copr repo for fmf owned by psss                 2.5 kB/s | 3.3 kB     00:01
Apr  3 08:39:02 rdma06 dnf[335085]: beaker-AppStream                                188 kB/s | 3.2 kB     00:00
Apr  3 08:39:02 rdma06 dnf[335085]: beaker-AppStream-debuginfo                      124 kB/s | 1.5 kB     00:00
Apr  3 08:39:03 rdma06 dnf[335085]: beaker-BaseOS                                   301 kB/s | 2.7 kB     00:00
Apr  3 08:39:03 rdma06 dnf[335085]: beaker-BaseOS-debuginfo                         163 kB/s | 1.5 kB     00:00
Apr  3 08:39:03 rdma06 dnf[335085]: beaker-CRB                                      241 kB/s | 2.7 kB     00:00
Apr  3 08:39:03 rdma06 dnf[335085]: beaker-CRB-debuginfo                            106 kB/s | 1.5 kB     00:00
Apr  3 08:39:03 rdma06 dnf[335085]: beaker-HighAvailability                         169 kB/s | 2.8 kB     00:00
Apr  3 08:39:03 rdma06 dnf[335085]: beaker-HighAvailability-debuginfo               153 kB/s | 1.5 kB     00:00
Apr  3 08:39:03 rdma06 dnf[335085]: beaker-NFV                                      168 kB/s | 2.7 kB     00:00
Apr  3 08:39:03 rdma06 dnf[335085]: beaker-NFV-debuginfo                            116 kB/s | 1.5 kB     00:00
Apr  3 08:39:03 rdma06 dnf[335085]: beaker-RT                                       216 kB/s | 2.7 kB     00:00
Apr  3 08:39:03 rdma06 dnf[335085]: beaker-RT-debuginfo                              85 kB/s | 1.5 kB     00:00
Apr  3 08:39:03 rdma06 dnf[335085]: beaker-ResilientStorage                         252 kB/s | 2.8 kB     00:00
Apr  3 08:39:03 rdma06 dnf[335085]: beaker-ResilientStorage-debuginfo                91 kB/s | 1.5 kB     00:00
Apr  3 08:39:03 rdma06 dnf[335085]: beaker-SAP                                      219 kB/s | 2.7 kB     00:00
Apr  3 08:39:03 rdma06 dnf[335085]: beaker-SAP-debuginfo                             75 kB/s | 1.5 kB     00:00
Apr  3 08:39:03 rdma06 dnf[335085]: beaker-SAPHANA                                  112 kB/s | 2.7 kB     00:00
Apr  3 08:39:03 rdma06 dnf[335085]: beaker-SAPHANA-debuginfo                        115 kB/s | 1.5 kB     00:00
Apr  3 08:39:04 rdma06 dnf[335085]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr  3 08:39:04 rdma06 dnf[335085]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr  3 08:39:05 rdma06 dnf[335085]: Metadata cache created.
Apr  3 08:39:05 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  3 08:39:05 rdma06 systemd[1]: Finished dnf makecache.
Apr  3 08:39:05 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.428s CPU time.
Apr  3 08:43:22 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 08:43:22 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 08:43:22 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 08:43:22 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 08:48:22 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 08:48:22 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 08:48:22 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 08:48:22 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 08:53:22 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 08:53:22 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 08:53:22 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 08:53:22 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 08:58:22 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 08:58:22 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 08:58:22 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 08:58:22 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 09:03:23 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 09:03:23 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 09:03:23 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 09:03:23 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 09:08:23 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 09:08:23 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 09:08:23 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 09:08:23 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 09:13:23 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 09:13:23 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 09:13:23 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 09:13:23 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 09:18:23 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 09:18:23 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 09:18:23 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 09:18:23 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 09:23:23 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 09:23:23 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 09:23:23 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 09:23:23 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 09:28:23 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 09:28:23 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 09:28:23 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 09:28:23 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 09:33:23 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 09:33:23 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 09:33:23 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 09:33:23 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 09:38:23 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 09:38:23 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 09:38:23 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 09:38:23 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 09:43:23 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 09:43:23 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 09:43:23 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 09:43:23 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 09:48:23 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 09:48:23 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 09:48:23 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 09:48:23 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 09:53:23 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 09:53:23 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 09:53:23 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 09:53:23 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 09:58:23 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 09:58:23 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 09:58:23 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 09:58:23 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 10:03:23 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 10:03:23 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 10:03:23 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 10:03:23 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 10:08:23 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 10:08:23 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 10:08:23 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 10:08:23 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 10:13:23 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 10:13:23 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 10:13:23 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 10:13:23 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 10:18:23 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 10:18:23 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 10:18:23 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 10:18:24 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 10:20:32 rdma06 NetworkManager[1025]: <info>  [1680531632.9089] dhcp4 (eno1): state changed new lease, address=10.73.131.45
Apr  3 10:23:24 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 10:23:24 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 10:23:24 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 10:23:24 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 10:24:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  3 10:24:01 rdma06 dnf[339343]: Updating Subscription Management repositories.
Apr  3 10:24:01 rdma06 dnf[339343]: Unable to read consumer identity
Apr  3 10:24:01 rdma06 dnf[339343]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  3 10:24:01 rdma06 dnf[339343]: Metadata cache refreshed recently.
Apr  3 10:24:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  3 10:24:01 rdma06 systemd[1]: Finished dnf makecache.
Apr  3 10:28:24 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 10:28:24 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 10:28:24 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 10:28:24 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 10:33:24 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 10:33:24 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 10:33:24 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 10:33:24 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 10:38:24 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 10:38:24 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 10:38:24 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 10:38:24 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 10:43:24 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 10:43:24 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 10:43:24 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 10:43:24 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 10:48:24 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 10:48:24 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 10:48:24 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 10:48:24 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 10:53:24 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 10:53:24 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 10:53:24 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 10:53:24 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 10:58:24 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 10:58:24 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 10:58:24 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 10:58:24 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 11:03:24 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 11:03:24 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 11:03:24 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 11:03:24 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 11:08:24 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 11:08:24 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 11:08:24 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 11:08:24 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 11:13:24 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 11:13:24 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 11:13:24 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 11:13:24 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 11:18:24 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 11:18:24 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 11:18:24 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 11:18:24 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 11:23:24 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 11:23:24 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 11:23:24 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 11:23:24 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 11:28:24 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 11:28:24 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 11:28:24 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 11:28:24 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 11:33:24 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 11:33:24 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 11:33:24 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 11:33:24 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 11:38:24 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 11:38:24 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 11:38:24 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 11:38:24 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 11:43:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  3 11:43:01 rdma06 dnf[342529]: Updating Subscription Management repositories.
Apr  3 11:43:01 rdma06 dnf[342529]: Unable to read consumer identity
Apr  3 11:43:01 rdma06 dnf[342529]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  3 11:43:02 rdma06 dnf[342529]: Copr repo for fmf owned by psss                 3.2 kB/s | 3.3 kB     00:01
Apr  3 11:43:02 rdma06 dnf[342529]: beaker-AppStream                                402 kB/s | 3.2 kB     00:00
Apr  3 11:43:02 rdma06 dnf[342529]: beaker-AppStream-debuginfo                      302 kB/s | 1.5 kB     00:00
Apr  3 11:43:02 rdma06 dnf[342529]: beaker-BaseOS                                   553 kB/s | 2.7 kB     00:00
Apr  3 11:43:02 rdma06 dnf[342529]: beaker-BaseOS-debuginfo                         331 kB/s | 1.5 kB     00:00
Apr  3 11:43:02 rdma06 dnf[342529]: beaker-CRB                                      116 kB/s | 2.7 kB     00:00
Apr  3 11:43:02 rdma06 dnf[342529]: beaker-CRB-debuginfo                            368 kB/s | 1.5 kB     00:00
Apr  3 11:43:02 rdma06 dnf[342529]: beaker-HighAvailability                         271 kB/s | 2.8 kB     00:00
Apr  3 11:43:02 rdma06 dnf[342529]: beaker-HighAvailability-debuginfo               381 kB/s | 1.5 kB     00:00
Apr  3 11:43:02 rdma06 dnf[342529]: beaker-NFV                                      607 kB/s | 2.7 kB     00:00
Apr  3 11:43:02 rdma06 dnf[342529]: beaker-NFV-debuginfo                             88 kB/s | 1.5 kB     00:00
Apr  3 11:43:02 rdma06 dnf[342529]: beaker-RT                                       389 kB/s | 2.7 kB     00:00
Apr  3 11:43:03 rdma06 dnf[342529]: beaker-RT-debuginfo                             389 kB/s | 1.5 kB     00:00
Apr  3 11:43:03 rdma06 dnf[342529]: beaker-ResilientStorage                         659 kB/s | 2.8 kB     00:00
Apr  3 11:43:03 rdma06 dnf[342529]: beaker-ResilientStorage-debuginfo               420 kB/s | 1.5 kB     00:00
Apr  3 11:43:03 rdma06 dnf[342529]: beaker-SAP                                      690 kB/s | 2.7 kB     00:00
Apr  3 11:43:03 rdma06 dnf[342529]: beaker-SAP-debuginfo                            395 kB/s | 1.5 kB     00:00
Apr  3 11:43:03 rdma06 dnf[342529]: beaker-SAPHANA                                  654 kB/s | 2.7 kB     00:00
Apr  3 11:43:03 rdma06 dnf[342529]: beaker-SAPHANA-debuginfo                        390 kB/s | 1.5 kB     00:00
Apr  3 11:43:03 rdma06 dnf[342529]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr  3 11:43:04 rdma06 dnf[342529]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr  3 11:43:04 rdma06 dnf[342529]: Metadata cache created.
Apr  3 11:43:04 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  3 11:43:04 rdma06 systemd[1]: Finished dnf makecache.
Apr  3 11:43:04 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.392s CPU time.
Apr  3 11:43:24 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 11:43:24 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 11:43:24 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 11:43:25 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 11:48:25 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 11:48:25 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 11:48:25 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 11:48:25 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 11:53:25 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 11:53:25 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 11:53:25 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 11:53:25 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 11:58:25 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 11:58:25 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 11:58:25 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 11:58:25 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 12:03:25 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 12:03:25 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 12:03:25 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 12:03:25 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 12:08:25 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 12:08:25 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 12:08:25 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 12:08:25 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 12:13:25 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 12:13:25 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 12:13:25 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 12:13:25 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 12:18:25 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 12:18:25 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 12:18:25 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 12:18:25 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 12:23:25 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 12:23:25 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 12:23:25 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 12:23:25 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 12:28:25 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 12:28:25 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 12:28:25 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 12:28:25 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 12:33:25 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 12:33:25 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 12:33:25 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 12:33:25 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 12:38:25 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 12:38:25 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 12:38:25 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 12:38:25 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 12:43:25 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 12:43:25 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 12:43:25 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 12:43:25 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 12:48:25 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 12:48:25 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 12:48:25 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 12:48:25 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 12:53:26 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 12:53:26 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 12:53:26 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 12:53:26 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.056 sec sweep
Apr  3 12:58:26 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 12:58:26 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 12:58:26 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 12:58:26 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 13:03:26 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 13:03:26 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 13:03:26 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 13:03:26 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 13:08:26 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 13:08:26 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 13:08:26 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 13:08:26 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 13:13:26 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 13:13:26 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 13:13:26 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 13:13:26 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 13:18:26 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 13:18:26 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 13:18:26 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 13:18:26 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 13:23:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  3 13:23:01 rdma06 dnf[346591]: Updating Subscription Management repositories.
Apr  3 13:23:01 rdma06 dnf[346591]: Unable to read consumer identity
Apr  3 13:23:01 rdma06 dnf[346591]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  3 13:23:01 rdma06 dnf[346591]: Metadata cache refreshed recently.
Apr  3 13:23:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  3 13:23:01 rdma06 systemd[1]: Finished dnf makecache.
Apr  3 13:23:26 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 13:23:26 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 13:23:26 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 13:23:26 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 13:28:26 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 13:28:26 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 13:28:26 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 13:28:26 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 13:33:26 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 13:33:26 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 13:33:26 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 13:33:26 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 13:38:26 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 13:38:26 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 13:38:26 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 13:38:26 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 13:43:27 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 13:43:27 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 13:43:27 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 13:43:27 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 13:48:27 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 13:48:27 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 13:48:27 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 13:48:27 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 13:53:27 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 13:53:27 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 13:53:27 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 13:53:27 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 13:58:27 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 13:58:27 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 13:58:27 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 13:58:27 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 14:03:27 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 14:03:27 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 14:03:27 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 14:03:27 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 14:08:27 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 14:08:27 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 14:08:27 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 14:08:27 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 14:13:27 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 14:13:27 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 14:13:27 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 14:13:27 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 14:18:27 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 14:18:27 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 14:18:27 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 14:18:27 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 14:23:27 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 14:23:27 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 14:23:27 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 14:23:27 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 14:28:27 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 14:28:27 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 14:28:27 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 14:28:27 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 14:33:28 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 14:33:28 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 14:33:28 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 14:33:28 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 14:38:28 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 14:38:28 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 14:38:28 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 14:38:28 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 14:43:28 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 14:43:28 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 14:43:28 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 14:43:28 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 14:48:28 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 14:48:28 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 14:48:28 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 14:48:28 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 14:53:28 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 14:53:28 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 14:53:28 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 14:53:28 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 14:58:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  3 14:58:01 rdma06 dnf[350418]: Updating Subscription Management repositories.
Apr  3 14:58:01 rdma06 dnf[350418]: Unable to read consumer identity
Apr  3 14:58:01 rdma06 dnf[350418]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  3 14:58:03 rdma06 dnf[350418]: Copr repo for fmf owned by psss                 2.2 kB/s | 3.3 kB     00:01
Apr  3 14:58:03 rdma06 dnf[350418]: beaker-AppStream                                217 kB/s | 3.2 kB     00:00
Apr  3 14:58:03 rdma06 dnf[350418]: beaker-AppStream-debuginfo                       45 kB/s | 1.5 kB     00:00
Apr  3 14:58:03 rdma06 dnf[350418]: beaker-BaseOS                                    19 kB/s | 2.7 kB     00:00
Apr  3 14:58:03 rdma06 dnf[350418]: beaker-BaseOS-debuginfo                         116 kB/s | 1.5 kB     00:00
Apr  3 14:58:03 rdma06 dnf[350418]: beaker-CRB                                      223 kB/s | 2.7 kB     00:00
Apr  3 14:58:03 rdma06 dnf[350418]: beaker-CRB-debuginfo                            101 kB/s | 1.5 kB     00:00
Apr  3 14:58:03 rdma06 dnf[350418]: beaker-HighAvailability                         488 kB/s | 2.8 kB     00:00
Apr  3 14:58:03 rdma06 dnf[350418]: beaker-HighAvailability-debuginfo               140 kB/s | 1.5 kB     00:00
Apr  3 14:58:03 rdma06 dnf[350418]: beaker-NFV                                      178 kB/s | 2.7 kB     00:00
Apr  3 14:58:03 rdma06 dnf[350418]: beaker-NFV-debuginfo                            164 kB/s | 1.5 kB     00:00
Apr  3 14:58:03 rdma06 dnf[350418]: beaker-RT                                       254 kB/s | 2.7 kB     00:00
Apr  3 14:58:03 rdma06 dnf[350418]: beaker-RT-debuginfo                             322 kB/s | 1.5 kB     00:00
Apr  3 14:58:03 rdma06 dnf[350418]: beaker-ResilientStorage                         320 kB/s | 2.8 kB     00:00
Apr  3 14:58:03 rdma06 dnf[350418]: beaker-ResilientStorage-debuginfo               154 kB/s | 1.5 kB     00:00
Apr  3 14:58:03 rdma06 dnf[350418]: beaker-SAP                                      197 kB/s | 2.7 kB     00:00
Apr  3 14:58:03 rdma06 dnf[350418]: beaker-SAP-debuginfo                            108 kB/s | 1.5 kB     00:00
Apr  3 14:58:03 rdma06 dnf[350418]: beaker-SAPHANA                                  577 kB/s | 2.7 kB     00:00
Apr  3 14:58:03 rdma06 dnf[350418]: beaker-SAPHANA-debuginfo                        103 kB/s | 1.5 kB     00:00
Apr  3 14:58:04 rdma06 dnf[350418]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr  3 14:58:05 rdma06 dnf[350418]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr  3 14:58:05 rdma06 dnf[350418]: Metadata cache created.
Apr  3 14:58:05 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  3 14:58:05 rdma06 systemd[1]: Finished dnf makecache.
Apr  3 14:58:05 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.387s CPU time.
Apr  3 14:58:28 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 14:58:28 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 14:58:28 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 14:58:28 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 15:03:28 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 15:03:28 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 15:03:28 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 15:03:28 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 15:08:28 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 15:08:28 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 15:08:28 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 15:08:28 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 15:13:28 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 15:13:28 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 15:13:28 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 15:13:28 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 15:18:28 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 15:18:28 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 15:18:28 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 15:18:28 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 15:23:28 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 15:23:28 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 15:23:28 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 15:23:29 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 15:28:29 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 15:28:29 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 15:28:29 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 15:28:29 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 15:33:29 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 15:33:29 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 15:33:29 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 15:33:29 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 15:38:29 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 15:38:29 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 15:38:29 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 15:38:29 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 15:43:29 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 15:43:29 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 15:43:29 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 15:43:29 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 15:48:29 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 15:48:29 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 15:48:29 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 15:48:29 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 15:53:29 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 15:53:29 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 15:53:29 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 15:53:29 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 15:58:29 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 15:58:29 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 15:58:29 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 15:58:29 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 16:03:29 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 16:03:29 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 16:03:29 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 16:03:29 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 16:08:29 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 16:08:29 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 16:08:29 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 16:08:29 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 16:09:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  3 16:09:01 rdma06 dnf[353315]: Updating Subscription Management repositories.
Apr  3 16:09:01 rdma06 dnf[353315]: Unable to read consumer identity
Apr  3 16:09:01 rdma06 dnf[353315]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  3 16:09:01 rdma06 dnf[353315]: Metadata cache refreshed recently.
Apr  3 16:09:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  3 16:09:01 rdma06 systemd[1]: Finished dnf makecache.
Apr  3 16:13:29 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 16:13:29 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 16:13:29 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 16:13:29 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 16:18:29 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 16:18:29 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 16:18:29 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 16:18:29 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 16:20:32 rdma06 NetworkManager[1025]: <info>  [1680553232.8751] dhcp4 (eno1): state changed new lease, address=10.73.131.45
Apr  3 16:23:29 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 16:23:29 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 16:23:29 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 16:23:29 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 16:28:29 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 16:28:29 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 16:28:29 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 16:28:29 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 16:33:29 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 16:33:29 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 16:33:29 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 16:33:29 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 16:38:29 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 16:38:29 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 16:38:29 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 16:38:29 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 16:43:29 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 16:43:29 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 16:43:29 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 16:43:29 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 16:48:29 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 16:48:29 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 16:48:29 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 16:48:29 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 16:53:30 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 16:53:30 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 16:53:30 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 16:53:30 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 16:58:30 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 16:58:30 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 16:58:30 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 16:58:30 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 17:03:30 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 17:03:30 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 17:03:30 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 17:03:30 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 17:08:30 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 17:08:30 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 17:08:30 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 17:08:30 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 17:13:30 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 17:13:30 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 17:13:30 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 17:13:30 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 17:18:30 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 17:18:30 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 17:18:30 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 17:18:30 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 17:23:30 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 17:23:30 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 17:23:30 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 17:23:30 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 17:28:30 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 17:28:30 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 17:28:30 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 17:28:30 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 17:33:30 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 17:33:30 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 17:33:30 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 17:33:30 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 17:38:30 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 17:38:30 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 17:38:30 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 17:38:30 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 17:41:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  3 17:41:01 rdma06 dnf[357021]: Updating Subscription Management repositories.
Apr  3 17:41:01 rdma06 dnf[357021]: Unable to read consumer identity
Apr  3 17:41:01 rdma06 dnf[357021]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  3 17:41:01 rdma06 dnf[357021]: Metadata cache refreshed recently.
Apr  3 17:41:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  3 17:41:01 rdma06 systemd[1]: Finished dnf makecache.
Apr  3 17:43:30 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 17:43:30 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 17:43:30 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 17:43:30 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 17:48:13 rdma06 chronyd[1034]: Selected source 10.13.199.1
Apr  3 17:48:30 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 17:48:30 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 17:48:30 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 17:48:30 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 17:53:30 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 17:53:30 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 17:53:30 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 17:53:30 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 17:58:30 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 17:58:30 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 17:58:30 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 17:58:30 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 18:03:30 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 18:03:30 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 18:03:30 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 18:03:30 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 18:08:30 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 18:08:30 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 18:08:30 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 18:08:31 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 18:13:31 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 18:13:31 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 18:13:31 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 18:13:31 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 18:18:31 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 18:18:31 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 18:18:31 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 18:18:31 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 18:23:31 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 18:23:31 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 18:23:31 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 18:23:31 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 18:28:31 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 18:28:31 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 18:28:31 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 18:28:31 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 18:33:31 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 18:33:31 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 18:33:31 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 18:33:31 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 18:38:31 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 18:38:31 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 18:38:31 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 18:38:31 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 18:43:31 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 18:43:31 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 18:43:31 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 18:43:31 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 18:48:31 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 18:48:31 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 18:48:31 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 18:48:31 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 18:53:31 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 18:53:31 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 18:53:31 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 18:53:31 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 18:58:31 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 18:58:31 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 18:58:31 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 18:58:31 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 19:01:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  3 19:01:01 rdma06 dnf[360248]: Updating Subscription Management repositories.
Apr  3 19:01:01 rdma06 dnf[360248]: Unable to read consumer identity
Apr  3 19:01:01 rdma06 dnf[360248]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  3 19:01:03 rdma06 dnf[360248]: Copr repo for fmf owned by psss                 1.8 kB/s | 3.3 kB     00:01
Apr  3 19:01:03 rdma06 dnf[360248]: beaker-AppStream                                639 kB/s | 3.2 kB     00:00
Apr  3 19:01:03 rdma06 dnf[360248]: beaker-AppStream-debuginfo                      405 kB/s | 1.5 kB     00:00
Apr  3 19:01:03 rdma06 dnf[360248]: beaker-BaseOS                                   672 kB/s | 2.7 kB     00:00
Apr  3 19:01:03 rdma06 dnf[360248]: beaker-BaseOS-debuginfo                         397 kB/s | 1.5 kB     00:00
Apr  3 19:01:03 rdma06 dnf[360248]: beaker-CRB                                      636 kB/s | 2.7 kB     00:00
Apr  3 19:01:03 rdma06 dnf[360248]: beaker-CRB-debuginfo                            402 kB/s | 1.5 kB     00:00
Apr  3 19:01:03 rdma06 dnf[360248]: beaker-HighAvailability                         667 kB/s | 2.8 kB     00:00
Apr  3 19:01:03 rdma06 dnf[360248]: beaker-HighAvailability-debuginfo               385 kB/s | 1.5 kB     00:00
Apr  3 19:01:03 rdma06 dnf[360248]: beaker-NFV                                      725 kB/s | 2.7 kB     00:00
Apr  3 19:01:03 rdma06 dnf[360248]: beaker-NFV-debuginfo                            419 kB/s | 1.5 kB     00:00
Apr  3 19:01:03 rdma06 dnf[360248]: beaker-RT                                       787 kB/s | 2.7 kB     00:00
Apr  3 19:01:03 rdma06 dnf[360248]: beaker-RT-debuginfo                             398 kB/s | 1.5 kB     00:00
Apr  3 19:01:03 rdma06 dnf[360248]: beaker-ResilientStorage                         679 kB/s | 2.8 kB     00:00
Apr  3 19:01:03 rdma06 dnf[360248]: beaker-ResilientStorage-debuginfo               391 kB/s | 1.5 kB     00:00
Apr  3 19:01:03 rdma06 dnf[360248]: beaker-SAP                                      718 kB/s | 2.7 kB     00:00
Apr  3 19:01:03 rdma06 dnf[360248]: beaker-SAP-debuginfo                            457 kB/s | 1.5 kB     00:00
Apr  3 19:01:03 rdma06 dnf[360248]: beaker-SAPHANA                                  754 kB/s | 2.7 kB     00:00
Apr  3 19:01:03 rdma06 dnf[360248]: beaker-SAPHANA-debuginfo                        481 kB/s | 1.5 kB     00:00
Apr  3 19:01:04 rdma06 dnf[360248]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr  3 19:01:05 rdma06 dnf[360248]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr  3 19:01:05 rdma06 dnf[360248]: Metadata cache created.
Apr  3 19:01:05 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  3 19:01:05 rdma06 systemd[1]: Finished dnf makecache.
Apr  3 19:01:05 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.426s CPU time.
Apr  3 19:03:31 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 19:03:31 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 19:03:31 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 19:03:31 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 19:08:31 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 19:08:31 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 19:08:31 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 19:08:31 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 19:13:31 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 19:13:31 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 19:13:31 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 19:13:31 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 19:18:31 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 19:18:31 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 19:18:31 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 19:18:31 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 19:23:31 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 19:23:31 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 19:23:31 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 19:23:31 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 19:28:31 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 19:28:31 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 19:28:31 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 19:28:32 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 19:33:32 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 19:33:32 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 19:33:32 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 19:33:32 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 19:38:32 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 19:38:32 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 19:38:32 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 19:38:32 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 19:43:32 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 19:43:32 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 19:43:32 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 19:43:32 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 19:48:32 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 19:48:32 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 19:48:32 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 19:48:32 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 19:53:32 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 19:53:32 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 19:53:32 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 19:53:32 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 19:58:32 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 19:58:32 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 19:58:32 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 19:58:32 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 20:03:32 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 20:03:32 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 20:03:32 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 20:03:32 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 20:08:32 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 20:08:32 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 20:08:32 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 20:08:32 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 20:13:32 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 20:13:32 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 20:13:32 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 20:13:32 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 20:18:32 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 20:18:32 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 20:18:32 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 20:18:32 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 20:23:32 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 20:23:32 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 20:23:32 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 20:23:32 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 20:28:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  3 20:28:01 rdma06 dnf[363777]: Updating Subscription Management repositories.
Apr  3 20:28:01 rdma06 dnf[363777]: Unable to read consumer identity
Apr  3 20:28:01 rdma06 dnf[363777]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  3 20:28:01 rdma06 dnf[363777]: Metadata cache refreshed recently.
Apr  3 20:28:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  3 20:28:01 rdma06 systemd[1]: Finished dnf makecache.
Apr  3 20:28:32 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 20:28:32 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 20:28:32 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 20:28:32 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 20:33:33 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 20:33:33 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 20:33:33 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 20:33:33 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 20:38:33 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 20:38:33 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 20:38:33 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 20:38:33 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 20:43:33 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 20:43:33 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 20:43:33 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 20:43:33 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 20:48:33 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 20:48:33 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 20:48:33 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 20:48:33 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 20:53:33 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 20:53:33 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 20:53:33 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 20:53:33 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 20:58:33 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 20:58:33 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 20:58:33 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 20:58:33 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 21:03:33 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 21:03:33 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 21:03:33 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 21:03:33 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 21:08:33 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 21:08:33 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 21:08:33 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 21:08:33 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 21:13:33 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 21:13:33 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 21:13:33 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 21:13:33 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 21:18:33 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 21:18:33 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 21:18:33 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 21:18:33 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 21:23:33 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 21:23:33 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 21:23:33 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 21:23:33 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 21:28:33 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 21:28:33 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 21:28:33 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 21:28:33 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 21:33:33 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 21:33:33 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 21:33:33 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 21:33:33 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 21:36:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  3 21:36:01 rdma06 dnf[366519]: Updating Subscription Management repositories.
Apr  3 21:36:01 rdma06 dnf[366519]: Unable to read consumer identity
Apr  3 21:36:01 rdma06 dnf[366519]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  3 21:36:01 rdma06 dnf[366519]: Metadata cache refreshed recently.
Apr  3 21:36:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  3 21:36:01 rdma06 systemd[1]: Finished dnf makecache.
Apr  3 21:38:33 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 21:38:33 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 21:38:33 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 21:38:34 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 21:43:34 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 21:43:34 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 21:43:34 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 21:43:34 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 21:48:34 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 21:48:34 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 21:48:34 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 21:48:34 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 21:53:34 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 21:53:34 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 21:53:34 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 21:53:34 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 21:58:34 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 21:58:34 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 21:58:34 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 21:58:34 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 22:03:34 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 22:03:34 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 22:03:34 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 22:03:34 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 22:08:34 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 22:08:34 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 22:08:34 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 22:08:34 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 22:13:34 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 22:13:34 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 22:13:34 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 22:13:34 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 22:18:34 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 22:18:34 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 22:18:34 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 22:18:34 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 22:20:32 rdma06 NetworkManager[1025]: <info>  [1680574832.8737] dhcp4 (eno1): state changed new lease, address=10.73.131.45
Apr  3 22:23:34 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 22:23:34 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 22:23:34 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 22:23:34 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 22:28:34 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 22:28:34 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 22:28:34 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 22:28:34 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 22:32:00 rdma06 systemd[5685]: Starting Cleanup of User's Temporary Files and Directories...
Apr  3 22:32:00 rdma06 systemd[5685]: Finished Cleanup of User's Temporary Files and Directories.
Apr  3 22:33:35 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 22:33:35 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 22:33:35 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 22:33:35 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 22:38:35 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 22:38:35 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 22:38:35 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 22:38:35 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 22:39:00 rdma06 systemd[1]: Starting Cleanup of Temporary Directories...
Apr  3 22:39:00 rdma06 systemd[1]: systemd-tmpfiles-clean.service: Deactivated successfully.
Apr  3 22:39:00 rdma06 systemd[1]: Finished Cleanup of Temporary Directories.
Apr  3 22:39:00 rdma06 systemd[1]: run-credentials-systemd\x2dtmpfiles\x2dclean.service.mount: Deactivated successfully.
Apr  3 22:43:35 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 22:43:35 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 22:43:35 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 22:43:35 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 22:48:35 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 22:48:35 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 22:48:35 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 22:48:35 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 22:53:35 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 22:53:35 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 22:53:35 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 22:53:35 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 22:58:35 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 22:58:35 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 22:58:35 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 22:58:35 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 23:03:35 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 23:03:35 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 23:03:35 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 23:03:35 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 23:08:35 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 23:08:35 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 23:08:35 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 23:08:35 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 23:13:35 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 23:13:35 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 23:13:35 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 23:13:35 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 23:18:35 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 23:18:35 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 23:18:35 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 23:18:36 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 23:20:50 rdma06 systemd[1]: Starting dnf makecache...
Apr  3 23:20:51 rdma06 dnf[370876]: Updating Subscription Management repositories.
Apr  3 23:20:51 rdma06 dnf[370876]: Unable to read consumer identity
Apr  3 23:20:51 rdma06 dnf[370876]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  3 23:20:52 rdma06 dnf[370876]: Copr repo for fmf owned by psss                 5.2 kB/s | 3.3 kB     00:00
Apr  3 23:20:52 rdma06 dnf[370876]: beaker-AppStream                                 50 kB/s | 3.2 kB     00:00
Apr  3 23:20:52 rdma06 dnf[370876]: beaker-AppStream-debuginfo                       40 kB/s | 1.5 kB     00:00
Apr  3 23:20:52 rdma06 dnf[370876]: beaker-BaseOS                                   115 kB/s | 2.7 kB     00:00
Apr  3 23:20:52 rdma06 dnf[370876]: beaker-BaseOS-debuginfo                          45 kB/s | 1.5 kB     00:00
Apr  3 23:20:52 rdma06 dnf[370876]: beaker-CRB                                       63 kB/s | 2.7 kB     00:00
Apr  3 23:20:52 rdma06 dnf[370876]: beaker-CRB-debuginfo                             67 kB/s | 1.5 kB     00:00
Apr  3 23:20:52 rdma06 dnf[370876]: beaker-HighAvailability                          77 kB/s | 2.8 kB     00:00
Apr  3 23:20:52 rdma06 dnf[370876]: beaker-HighAvailability-debuginfo                48 kB/s | 1.5 kB     00:00
Apr  3 23:20:52 rdma06 dnf[370876]: beaker-NFV                                       34 kB/s | 2.7 kB     00:00
Apr  3 23:20:52 rdma06 dnf[370876]: beaker-NFV-debuginfo                             56 kB/s | 1.5 kB     00:00
Apr  3 23:20:52 rdma06 dnf[370876]: beaker-RT                                        55 kB/s | 2.7 kB     00:00
Apr  3 23:20:53 rdma06 dnf[370876]: beaker-RT-debuginfo                              34 kB/s | 1.5 kB     00:00
Apr  3 23:20:53 rdma06 dnf[370876]: beaker-ResilientStorage                          59 kB/s | 2.8 kB     00:00
Apr  3 23:20:53 rdma06 dnf[370876]: beaker-ResilientStorage-debuginfo                59 kB/s | 1.5 kB     00:00
Apr  3 23:20:53 rdma06 dnf[370876]: beaker-SAP                                      152 kB/s | 2.7 kB     00:00
Apr  3 23:20:53 rdma06 dnf[370876]: beaker-SAP-debuginfo                             31 kB/s | 1.5 kB     00:00
Apr  3 23:20:53 rdma06 dnf[370876]: beaker-SAPHANA                                   36 kB/s | 2.7 kB     00:00
Apr  3 23:20:53 rdma06 dnf[370876]: beaker-SAPHANA-debuginfo                         33 kB/s | 1.5 kB     00:00
Apr  3 23:20:53 rdma06 dnf[370876]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr  3 23:20:54 rdma06 dnf[370876]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr  3 23:20:54 rdma06 dnf[370876]: Metadata cache created.
Apr  3 23:20:54 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  3 23:20:54 rdma06 systemd[1]: Finished dnf makecache.
Apr  3 23:20:54 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.421s CPU time.
Apr  3 23:23:36 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 23:23:36 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 23:23:36 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 23:23:36 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 23:28:36 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 23:28:36 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 23:28:36 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 23:28:36 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 23:33:36 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 23:33:36 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 23:33:36 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 23:33:36 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 23:38:36 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 23:38:36 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 23:38:36 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 23:38:36 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 23:43:36 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 23:43:36 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 23:43:36 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 23:43:36 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 23:48:36 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 23:48:36 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 23:48:36 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 23:48:36 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  3 23:53:36 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 23:53:36 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 23:53:36 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 23:53:36 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  3 23:58:36 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  3 23:58:36 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  3 23:58:36 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  3 23:58:36 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 00:01:00 rdma06 systemd[1]: Starting Rotate log files...
Apr  4 00:01:00 rdma06 systemd[1]: logrotate.service: Deactivated successfully.
Apr  4 00:01:00 rdma06 systemd[1]: Finished Rotate log files.
Apr  4 00:03:36 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 00:03:36 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 00:03:36 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 00:03:36 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  4 00:08:36 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 00:08:36 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 00:08:36 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 00:08:37 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 00:13:37 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 00:13:37 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 00:13:37 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 00:13:37 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 00:18:37 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 00:18:37 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 00:18:37 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 00:18:37 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 00:23:37 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 00:23:37 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 00:23:37 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 00:23:37 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  4 00:28:37 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 00:28:37 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 00:28:37 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 00:28:37 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 00:33:37 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 00:33:37 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 00:33:37 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 00:33:37 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 00:38:37 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 00:38:37 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 00:38:37 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 00:38:37 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  4 00:40:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  4 00:40:01 rdma06 dnf[374092]: Updating Subscription Management repositories.
Apr  4 00:40:01 rdma06 dnf[374092]: Unable to read consumer identity
Apr  4 00:40:01 rdma06 dnf[374092]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  4 00:40:01 rdma06 dnf[374092]: Metadata cache refreshed recently.
Apr  4 00:40:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  4 00:40:01 rdma06 systemd[1]: Finished dnf makecache.
Apr  4 00:43:37 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 00:43:37 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 00:43:37 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 00:43:37 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 00:48:37 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 00:48:37 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 00:48:37 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 00:48:37 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  4 00:53:37 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 00:53:37 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 00:53:37 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 00:53:37 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 00:58:37 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 00:58:37 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 00:58:37 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 00:58:37 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 01:03:38 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 01:03:38 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 01:03:38 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 01:03:38 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  4 01:08:38 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 01:08:38 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 01:08:38 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 01:08:38 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 01:13:38 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 01:13:38 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 01:13:38 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 01:13:38 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 01:18:38 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 01:18:38 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 01:18:38 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 01:18:38 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 01:23:38 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 01:23:38 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 01:23:38 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 01:23:38 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 01:28:38 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 01:28:38 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 01:28:38 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 01:28:38 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 01:33:38 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 01:33:38 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 01:33:38 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 01:33:38 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 01:38:38 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 01:38:38 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 01:38:38 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 01:38:38 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 01:39:01 rdma06 chronyd[1034]: Selected source 10.11.160.238
Apr  4 01:43:38 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 01:43:38 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 01:43:38 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 01:43:38 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 01:48:38 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 01:48:38 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 01:48:38 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 01:48:38 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  4 01:53:38 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 01:53:38 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 01:53:38 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 01:53:39 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 01:55:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  4 01:55:01 rdma06 dnf[377117]: Updating Subscription Management repositories.
Apr  4 01:55:01 rdma06 dnf[377117]: Unable to read consumer identity
Apr  4 01:55:01 rdma06 dnf[377117]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  4 01:55:01 rdma06 dnf[377117]: Metadata cache refreshed recently.
Apr  4 01:55:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  4 01:55:01 rdma06 systemd[1]: Finished dnf makecache.
Apr  4 01:58:39 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 01:58:39 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 01:58:39 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 01:58:39 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 02:03:39 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 02:03:39 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 02:03:39 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 02:03:39 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  4 02:08:39 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 02:08:39 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 02:08:39 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 02:08:39 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 02:13:39 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 02:13:39 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 02:13:39 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 02:13:39 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 02:18:39 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 02:18:39 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 02:18:39 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 02:18:39 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  4 02:23:39 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 02:23:39 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 02:23:39 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 02:23:39 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 02:28:39 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 02:28:39 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 02:28:39 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 02:28:39 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 02:33:39 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 02:33:39 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 02:33:39 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 02:33:39 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 02:38:39 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 02:38:39 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 02:38:39 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 02:38:39 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 02:43:39 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 02:43:39 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 02:43:39 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 02:43:40 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 02:48:40 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 02:48:40 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 02:48:40 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 02:48:40 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 02:53:40 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 02:53:40 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 02:53:40 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 02:53:40 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 02:58:40 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 02:58:40 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 02:58:40 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 02:58:40 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 03:03:40 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 03:03:40 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 03:03:40 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 03:03:40 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  4 03:08:40 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 03:08:40 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 03:08:40 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 03:08:40 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  4 03:13:40 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 03:13:40 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 03:13:40 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 03:13:40 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 03:18:40 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 03:18:40 rdma06 fm0_sm[138306]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 03:18:40 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 03:18:40 rdma06 fm0_sm[138306]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.056 sec sweep
Apr  4 03:22:05 rdma06 systemd[1]: Stopping OPA Fabric Manager...
Apr  4 03:22:05 rdma06 opafmd[138293]: Stopped instance 0 of SM.
Apr  4 03:22:05 rdma06 fm0_sm[138306]: PROGR[main]: SM: sm_control_shutdown: FM exiting.
Apr  4 03:22:05 rdma06 systemd[1]: opafm.service: Deactivated successfully.
Apr  4 03:22:05 rdma06 systemd[1]: Stopped OPA Fabric Manager.
Apr  4 03:22:05 rdma06 systemd[1]: opafm.service: Consumed 17min 16.108s CPU time.
Apr  4 03:22:05 rdma06 systemd[1]: Starting OPA Fabric Manager...
Apr  4 03:22:05 rdma06 systemd[1]: Started OPA Fabric Manager.
Apr  4 03:22:05 rdma06 opafmd[380779]: WARN [main]: APP: XmlParsePrintWarning: Parse warning at line 2293 in tag 'Fm': VirtualFabric configuration:: Multicast group matching XML parse error - Implicit MC Groups did not match any enabled VF.
Apr  4 03:22:05 rdma06 opafmd[380779]: Parse warning at line 2293 in tag 'Fm': VirtualFabric configuration:: Multicast group matching XML parse error - Implicit MC Groups did not match any enabled VF.
Apr  4 03:22:05 rdma06 opafmd[380779]: Started instance 0 of SM.
Apr  4 03:22:05 rdma06 fm0_sm[380791]: [main]: Subnet Manager starting up. LogLevel: 2 LogMode: 0
Apr  4 03:22:05 rdma06 fm0_sm[380791]: [main]: SM: Version: 10.11.2.0.3
Apr  4 03:22:05 rdma06 fm0_sm[380791]: [main]: SM: Enabling CoreDumps to /var/crash/opafm up to 10G bytes
Apr  4 03:22:05 rdma06 fm0_sm[380791]: WARN [main]: SM: RenderVirtualFabrics: VirtualFabric configuration:: Multicast group matching XML parse error - Implicit MC Groups did not match any enabled VF.
Apr  4 03:22:05 rdma06 fm0_sm[380791]: [main]: SM: Using dynamic packet lifetime values 14, 15, 15, 16, 16, 16, 16, 17, 17
Apr  4 03:22:05 rdma06 fm0_sm[380791]: [main]: SM: GidPrefix=0xfe80000000000000, Key=0x0000000000000000, MKey=0x0000000000000000 : protect_level=1 : lease=0 seconds, dbsync interval=900 seconds
Apr  4 03:22:05 rdma06 fm0_sm[380791]: [main]: SM: Size Limits: EndNodePorts=9216 Nodes=11904 Ports=76416 Links=55296
Apr  4 03:22:05 rdma06 fm0_sm[380791]: [main]: SM: Memory: Pool=262144K SA Resp=38208K
Apr  4 03:22:05 rdma06 fm0_sm[380791]: PROGR[main]: SM: sm_main: Routing Algorithm in use: shortestpath
Apr  4 03:22:05 rdma06 fm0_sm[380791]: PROGR[main]: SM: sm_routing_func_assign_sls: 2 QOSGroups require 2 SLs and 2 SCs for operation
Apr  4 03:22:05 rdma06 fm0_sm[380791]: PROGR[main]: SM: sm_update_bw: QOSGroup Bandwidth Allocations : Total allocated bandwidth is 100%
Apr  4 03:22:05 rdma06 fm0_sm[380791]: PROGR[main]: SM: sm_update_bw: QoS(LowPriority) shared by 1 VFs; Assigned 100% Bandwidth on BaseSL 0
Apr  4 03:22:05 rdma06 fm0_sm[380791]: PROGR[main]: SM: sm_update_bw: QoS(Tenant_QOS) shared by 0 VFs; Assigned 0% Bandwidth on BaseSL 1
Apr  4 03:22:05 rdma06 fm0_sm[380791]: PROGR[main]: SM: [VF:Admin] : Base SL:0 Resp SL:0 Requires Resp SL:0 Multicast SL:0 QOS:1 HP:0 PKey:0x7fff
Apr  4 03:22:06 rdma06 fm0_sm[380791]: [main]: SM: Using: HFI 1 Port 1 PortGuid 0x001175010174479a
Apr  4 03:22:09 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_transition: SM STATE TRANSITION from NOTACTIVE to DISCOVERING
Apr  4 03:22:09 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Initial sweep.
Apr  4 03:22:09 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 03:22:09 rdma06 fm0_sm[380791]: rdma06.rhts.eng.pek2.redhat.com; MSG:NOTICE|SM:rdma06.rhts.eng.pek2.redhat.com:port 1|COND:#5 SM state to master|NODE:rdma06.rhts.eng.pek2.redhat.com:port 1:0x001175010174479a|DETAIL:transition from DISCOVERING to MASTER
Apr  4 03:22:09 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 15 packets, 0 retries, 0.074 sec sweep
Apr  4 03:22:09 rdma06 fm0_sm[380791]: rdma06.rhts.eng.pek2.redhat.com; MSG:INFORMATION|SM:rdma06.rhts.eng.pek2.redhat.com:port 1|COND:#12 Fabric Summary|NODE:rdma06.rhts.eng.pek2.redhat.com:port 1:0x001175010174479a|DETAIL:0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s)
Apr  4 03:22:09 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: SM redundancy not available
Apr  4 03:22:09 rdma06 fm0_sm[380791]: PROGR[async]: SA: sa_Trap: Received an (IS_SM off) CAPABILITYMASK CHANGE [0x00490020] trap from rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1, TID=0x0000000000000017
Apr  4 03:22:10 rdma06 fm0_sm[380791]: [pm]: Performance Manager starting up. LogLevel: 2 LogMode: 0
Apr  4 03:22:10 rdma06 fm0_sm[380791]: [pm]: Size Limits: EndNodePorts=9216
Apr  4 03:22:10 rdma06 fm0_sm[380791]: [pm]: Memory: Pool=3896585K
Apr  4 03:22:10 rdma06 fm0_sm[380791]: [pm]: Using: HFI 1 Port 1 PortGuid 0x001175010174479a
Apr  4 03:22:10 rdma06 fm0_sm[380791]: PROGR[async]: SA: sa_Trap: Received an (IS_SM on) CAPABILITYMASK CHANGE [0x00490022] trap from rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1, TID=0x0000000000000018
Apr  4 03:22:11 rdma06 fm0_sm[380791]: PROGR[pm]: IF3: rmpp_protocol_init: Allocating context pool with num entries=18432
Apr  4 03:22:12 rdma06 fm0_sm[380791]: [PmEngine]: PM: Engine starting up, will be monitoring and clearing port counters
Apr  4 03:22:12 rdma06 fm0_sm[380791]: [PmDbsyncThread]: PM: Dbsync thread starting up, will be syncing RAM/Disk PA history data
Apr  4 03:27:09 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 03:27:09 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 03:27:09 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 03:27:09 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  4 03:32:09 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 03:32:09 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 03:32:09 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 03:32:09 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  4 03:37:09 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 03:37:09 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 03:37:09 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 03:37:09 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  4 03:42:09 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 03:42:09 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 03:42:09 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 03:42:09 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  4 03:47:09 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 03:47:09 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 03:47:09 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 03:47:09 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.056 sec sweep
Apr  4 03:50:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  4 03:50:01 rdma06 dnf[382037]: Updating Subscription Management repositories.
Apr  4 03:50:01 rdma06 dnf[382037]: Unable to read consumer identity
Apr  4 03:50:01 rdma06 dnf[382037]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  4 03:50:03 rdma06 dnf[382037]: Copr repo for fmf owned by psss                 1.8 kB/s | 3.3 kB     00:01
Apr  4 03:50:03 rdma06 dnf[382037]: beaker-AppStream                                 74 kB/s | 3.2 kB     00:00
Apr  4 03:50:03 rdma06 dnf[382037]: beaker-AppStream-debuginfo                       49 kB/s | 1.5 kB     00:00
Apr  4 03:50:03 rdma06 dnf[382037]: beaker-BaseOS                                    68 kB/s | 2.7 kB     00:00
Apr  4 03:50:03 rdma06 dnf[382037]: beaker-BaseOS-debuginfo                          45 kB/s | 1.5 kB     00:00
Apr  4 03:50:03 rdma06 dnf[382037]: beaker-CRB                                       60 kB/s | 2.7 kB     00:00
Apr  4 03:50:03 rdma06 dnf[382037]: beaker-CRB-debuginfo                             79 kB/s | 1.5 kB     00:00
Apr  4 03:50:03 rdma06 dnf[382037]: beaker-HighAvailability                          61 kB/s | 2.8 kB     00:00
Apr  4 03:50:03 rdma06 dnf[382037]: beaker-HighAvailability-debuginfo                42 kB/s | 1.5 kB     00:00
Apr  4 03:50:03 rdma06 dnf[382037]: beaker-NFV                                       66 kB/s | 2.7 kB     00:00
Apr  4 03:50:03 rdma06 dnf[382037]: beaker-NFV-debuginfo                             52 kB/s | 1.5 kB     00:00
Apr  4 03:50:04 rdma06 dnf[382037]: beaker-RT                                        59 kB/s | 2.7 kB     00:00
Apr  4 03:50:04 rdma06 dnf[382037]: beaker-RT-debuginfo                              70 kB/s | 1.5 kB     00:00
Apr  4 03:50:04 rdma06 dnf[382037]: beaker-ResilientStorage                          71 kB/s | 2.8 kB     00:00
Apr  4 03:50:04 rdma06 dnf[382037]: beaker-ResilientStorage-debuginfo                42 kB/s | 1.5 kB     00:00
Apr  4 03:50:04 rdma06 dnf[382037]: beaker-SAP                                       81 kB/s | 2.7 kB     00:00
Apr  4 03:50:04 rdma06 dnf[382037]: beaker-SAP-debuginfo                             76 kB/s | 1.5 kB     00:00
Apr  4 03:50:04 rdma06 dnf[382037]: beaker-SAPHANA                                   64 kB/s | 2.7 kB     00:00
Apr  4 03:50:04 rdma06 dnf[382037]: beaker-SAPHANA-debuginfo                         39 kB/s | 1.5 kB     00:00
Apr  4 03:50:04 rdma06 dnf[382037]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr  4 03:50:05 rdma06 dnf[382037]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr  4 03:50:05 rdma06 dnf[382037]: Metadata cache created.
Apr  4 03:50:05 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  4 03:50:05 rdma06 systemd[1]: Finished dnf makecache.
Apr  4 03:50:05 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.369s CPU time.
Apr  4 03:52:09 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 03:52:09 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 03:52:09 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 03:52:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.056 sec sweep
Apr  4 03:57:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 03:57:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 03:57:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 03:57:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  4 04:02:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 04:02:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 04:02:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 04:02:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 04:07:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 04:07:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 04:07:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 04:07:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  4 04:12:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 04:12:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 04:12:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 04:12:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  4 04:17:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 04:17:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 04:17:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 04:17:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 04:20:32 rdma06 NetworkManager[1025]: <info>  [1680596432.9081] dhcp4 (eno1): state changed new lease, address=10.73.131.45
Apr  4 04:22:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 04:22:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 04:22:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 04:22:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 04:27:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 04:27:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 04:27:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 04:27:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  4 04:32:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 04:32:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 04:32:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 04:32:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 04:37:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 04:37:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 04:37:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 04:37:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 04:42:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 04:42:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 04:42:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 04:42:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  4 04:42:21 rdma06 journal[2657]: Adjustment to local watchdog ignored since 'no_localwatchdog' metadata is set
Apr  4 04:47:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 04:47:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 04:47:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 04:47:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  4 04:52:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 04:52:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 04:52:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 04:52:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 04:57:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 04:57:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 04:57:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 04:57:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  4 05:02:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 05:02:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 05:02:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 05:02:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  4 05:02:49 rdma06 chronyd[1034]: Selected source 10.18.100.10
Apr  4 05:07:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 05:07:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 05:07:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 05:07:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 05:12:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 05:12:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 05:12:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 05:12:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 05:17:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 05:17:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 05:17:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 05:17:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  4 05:22:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 05:22:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 05:22:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 05:22:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 05:27:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 05:27:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 05:27:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 05:27:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  4 05:32:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 05:32:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 05:32:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 05:32:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 05:37:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 05:37:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 05:37:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 05:37:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 05:41:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  4 05:41:01 rdma06 dnf[386613]: Updating Subscription Management repositories.
Apr  4 05:41:01 rdma06 dnf[386613]: Unable to read consumer identity
Apr  4 05:41:01 rdma06 dnf[386613]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  4 05:41:01 rdma06 dnf[386613]: Metadata cache refreshed recently.
Apr  4 05:41:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  4 05:41:01 rdma06 systemd[1]: Finished dnf makecache.
Apr  4 05:42:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 05:42:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 05:42:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 05:42:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  4 05:47:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 05:47:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 05:47:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 05:47:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 05:52:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 05:52:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 05:52:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 05:52:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 05:57:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 05:57:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 05:57:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 05:57:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 06:02:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 06:02:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 06:02:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 06:02:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 06:07:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 06:07:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 06:07:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 06:07:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 06:12:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 06:12:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 06:12:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 06:12:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 06:17:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 06:17:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 06:17:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 06:17:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 06:22:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 06:22:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 06:22:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 06:22:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  4 06:27:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 06:27:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 06:27:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 06:27:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 06:32:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 06:32:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 06:32:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 06:32:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 06:37:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 06:37:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 06:37:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 06:37:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 06:42:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 06:42:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 06:42:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 06:42:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 06:47:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 06:47:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 06:47:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 06:47:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 06:52:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 06:52:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 06:52:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 06:52:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 06:57:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 06:57:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 06:57:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 06:57:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 07:02:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 07:02:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 07:02:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 07:02:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 07:07:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 07:07:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 07:07:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 07:07:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 07:12:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 07:12:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 07:12:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 07:12:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 07:17:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 07:17:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 07:17:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 07:17:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 07:22:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 07:22:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 07:22:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 07:22:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 07:26:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  4 07:26:01 rdma06 dnf[390854]: Updating Subscription Management repositories.
Apr  4 07:26:01 rdma06 dnf[390854]: Unable to read consumer identity
Apr  4 07:26:01 rdma06 dnf[390854]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  4 07:26:03 rdma06 dnf[390854]: Copr repo for fmf owned by psss                 1.7 kB/s | 3.3 kB     00:01
Apr  4 07:26:03 rdma06 dnf[390854]: beaker-AppStream                                 67 kB/s | 3.2 kB     00:00
Apr  4 07:26:03 rdma06 dnf[390854]: beaker-AppStream-debuginfo                       30 kB/s | 1.5 kB     00:00
Apr  4 07:26:03 rdma06 dnf[390854]: beaker-BaseOS                                    56 kB/s | 2.7 kB     00:00
Apr  4 07:26:03 rdma06 dnf[390854]: beaker-BaseOS-debuginfo                          39 kB/s | 1.5 kB     00:00
Apr  4 07:26:03 rdma06 dnf[390854]: beaker-CRB                                       56 kB/s | 2.7 kB     00:00
Apr  4 07:26:03 rdma06 dnf[390854]: beaker-CRB-debuginfo                            102 kB/s | 1.5 kB     00:00
Apr  4 07:26:03 rdma06 dnf[390854]: beaker-HighAvailability                          76 kB/s | 2.8 kB     00:00
Apr  4 07:26:04 rdma06 dnf[390854]: beaker-HighAvailability-debuginfo                41 kB/s | 1.5 kB     00:00
Apr  4 07:26:04 rdma06 dnf[390854]: beaker-NFV                                       76 kB/s | 2.7 kB     00:00
Apr  4 07:26:04 rdma06 dnf[390854]: beaker-NFV-debuginfo                             49 kB/s | 1.5 kB     00:00
Apr  4 07:26:04 rdma06 dnf[390854]: beaker-RT                                        57 kB/s | 2.7 kB     00:00
Apr  4 07:26:04 rdma06 dnf[390854]: beaker-RT-debuginfo                              25 kB/s | 1.5 kB     00:00
Apr  4 07:26:04 rdma06 dnf[390854]: beaker-ResilientStorage                          64 kB/s | 2.8 kB     00:00
Apr  4 07:26:04 rdma06 dnf[390854]: beaker-ResilientStorage-debuginfo                85 kB/s | 1.5 kB     00:00
Apr  4 07:26:04 rdma06 dnf[390854]: beaker-SAP                                      105 kB/s | 2.7 kB     00:00
Apr  4 07:26:04 rdma06 dnf[390854]: beaker-SAP-debuginfo                             59 kB/s | 1.5 kB     00:00
Apr  4 07:26:04 rdma06 dnf[390854]: beaker-SAPHANA                                   65 kB/s | 2.7 kB     00:00
Apr  4 07:26:04 rdma06 dnf[390854]: beaker-SAPHANA-debuginfo                         34 kB/s | 1.5 kB     00:00
Apr  4 07:26:05 rdma06 dnf[390854]: beaker-harness                                  1.9 kB/s | 1.3 kB     00:00
Apr  4 07:26:05 rdma06 dnf[390854]: beaker-tasks                                    1.8 kB/s | 1.3 kB     00:00
Apr  4 07:26:06 rdma06 dnf[390854]: Metadata cache created.
Apr  4 07:26:06 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  4 07:26:06 rdma06 systemd[1]: Finished dnf makecache.
Apr  4 07:26:06 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.409s CPU time.
Apr  4 07:27:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 07:27:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 07:27:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 07:27:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  4 07:32:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 07:32:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 07:32:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 07:32:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  4 07:37:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 07:37:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 07:37:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 07:37:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 07:42:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 07:42:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 07:42:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 07:42:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 07:47:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 07:47:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 07:47:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 07:47:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  4 07:52:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 07:52:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 07:52:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 07:52:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 07:57:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 07:57:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 07:57:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 07:57:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 08:02:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 08:02:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 08:02:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 08:02:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 08:07:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 08:07:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 08:07:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 08:07:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 08:12:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 08:12:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 08:12:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 08:12:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 08:17:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 08:17:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 08:17:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 08:17:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 08:22:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 08:22:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 08:22:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 08:22:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  4 08:27:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 08:27:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 08:27:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 08:27:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 08:29:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  4 08:29:01 rdma06 dnf[393421]: Updating Subscription Management repositories.
Apr  4 08:29:01 rdma06 dnf[393421]: Unable to read consumer identity
Apr  4 08:29:01 rdma06 dnf[393421]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  4 08:29:01 rdma06 dnf[393421]: Metadata cache refreshed recently.
Apr  4 08:29:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  4 08:29:01 rdma06 systemd[1]: Finished dnf makecache.
Apr  4 08:32:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 08:32:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 08:32:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 08:32:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.056 sec sweep
Apr  4 08:37:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 08:37:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 08:37:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 08:37:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 08:42:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 08:42:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 08:42:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 08:42:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 08:47:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 08:47:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 08:47:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 08:47:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 08:52:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 08:52:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 08:52:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 08:52:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 08:57:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 08:57:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 08:57:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 08:57:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  4 09:02:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 09:02:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 09:02:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 09:02:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 09:07:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 09:07:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 09:07:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 09:07:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 09:10:35 rdma06 kernel: perf: interrupt took too long (4942 > 4935), lowering kernel.perf_event_max_sample_rate to 40000
Apr  4 09:12:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 09:12:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 09:12:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 09:12:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 09:17:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 09:17:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 09:17:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 09:17:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  4 09:19:13 rdma06 chronyd[1034]: Selected source 10.13.199.1
Apr  4 09:22:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 09:22:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 09:22:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 09:22:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 09:27:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 09:27:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 09:27:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 09:27:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 09:32:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 09:32:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 09:32:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 09:32:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 09:37:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 09:37:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 09:37:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 09:37:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 09:38:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  4 09:38:01 rdma06 dnf[396213]: Updating Subscription Management repositories.
Apr  4 09:38:01 rdma06 dnf[396213]: Unable to read consumer identity
Apr  4 09:38:01 rdma06 dnf[396213]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  4 09:38:01 rdma06 dnf[396213]: Metadata cache refreshed recently.
Apr  4 09:38:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  4 09:38:01 rdma06 systemd[1]: Finished dnf makecache.
Apr  4 09:42:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 09:42:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 09:42:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 09:42:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  4 09:47:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 09:47:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 09:47:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 09:47:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 09:52:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 09:52:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 09:52:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 09:52:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 09:57:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 09:57:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 09:57:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 09:57:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 10:02:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 10:02:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 10:02:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 10:02:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 10:07:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 10:07:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 10:07:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 10:07:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 10:12:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 10:12:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 10:12:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 10:12:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 10:17:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 10:17:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 10:17:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 10:17:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 10:20:32 rdma06 NetworkManager[1025]: <info>  [1680618032.9209] dhcp4 (eno1): state changed new lease, address=10.73.131.45
Apr  4 10:22:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 10:22:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 10:22:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 10:22:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  4 10:27:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 10:27:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 10:27:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 10:27:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 10:32:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 10:32:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 10:32:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 10:32:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 10:37:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 10:37:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 10:37:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 10:37:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 10:37:55 rdma06 chronyd[1034]: Selected source 10.11.160.238
Apr  4 10:42:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 10:42:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 10:42:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 10:42:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  4 10:47:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 10:47:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 10:47:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 10:47:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 10:52:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 10:52:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 10:52:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 10:52:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 10:57:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 10:57:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 10:57:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 10:57:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 11:02:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 11:02:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 11:02:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 11:02:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 11:07:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 11:07:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 11:07:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 11:07:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  4 11:12:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 11:12:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 11:12:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 11:12:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 11:16:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  4 11:16:01 rdma06 dnf[400177]: Updating Subscription Management repositories.
Apr  4 11:16:01 rdma06 dnf[400177]: Unable to read consumer identity
Apr  4 11:16:01 rdma06 dnf[400177]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  4 11:16:02 rdma06 dnf[400177]: Copr repo for fmf owned by psss                 2.7 kB/s | 3.3 kB     00:01
Apr  4 11:16:02 rdma06 dnf[400177]: beaker-AppStream                                 73 kB/s | 3.2 kB     00:00
Apr  4 11:16:02 rdma06 dnf[400177]: beaker-AppStream-debuginfo                       43 kB/s | 1.5 kB     00:00
Apr  4 11:16:03 rdma06 dnf[400177]: beaker-BaseOS                                   135 kB/s | 2.7 kB     00:00
Apr  4 11:16:03 rdma06 dnf[400177]: beaker-BaseOS-debuginfo                          35 kB/s | 1.5 kB     00:00
Apr  4 11:16:03 rdma06 dnf[400177]: beaker-CRB                                       70 kB/s | 2.7 kB     00:00
Apr  4 11:16:03 rdma06 dnf[400177]: beaker-CRB-debuginfo                             90 kB/s | 1.5 kB     00:00
Apr  4 11:16:03 rdma06 dnf[400177]: beaker-HighAvailability                          67 kB/s | 2.8 kB     00:00
Apr  4 11:16:03 rdma06 dnf[400177]: beaker-HighAvailability-debuginfo                40 kB/s | 1.5 kB     00:00
Apr  4 11:16:03 rdma06 dnf[400177]: beaker-NFV                                       69 kB/s | 2.7 kB     00:00
Apr  4 11:16:03 rdma06 dnf[400177]: beaker-NFV-debuginfo                             49 kB/s | 1.5 kB     00:00
Apr  4 11:16:03 rdma06 dnf[400177]: beaker-RT                                        57 kB/s | 2.7 kB     00:00
Apr  4 11:16:03 rdma06 dnf[400177]: beaker-RT-debuginfo                              28 kB/s | 1.5 kB     00:00
Apr  4 11:16:03 rdma06 dnf[400177]: beaker-ResilientStorage                          56 kB/s | 2.8 kB     00:00
Apr  4 11:16:03 rdma06 dnf[400177]: beaker-ResilientStorage-debuginfo                49 kB/s | 1.5 kB     00:00
Apr  4 11:16:03 rdma06 dnf[400177]: beaker-SAP                                       69 kB/s | 2.7 kB     00:00
Apr  4 11:16:03 rdma06 dnf[400177]: beaker-SAP-debuginfo                             59 kB/s | 1.5 kB     00:00
Apr  4 11:16:03 rdma06 dnf[400177]: beaker-SAPHANA                                   45 kB/s | 2.7 kB     00:00
Apr  4 11:16:03 rdma06 dnf[400177]: beaker-SAPHANA-debuginfo                         30 kB/s | 1.5 kB     00:00
Apr  4 11:16:04 rdma06 dnf[400177]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr  4 11:16:05 rdma06 dnf[400177]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr  4 11:16:05 rdma06 dnf[400177]: Metadata cache created.
Apr  4 11:16:05 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  4 11:16:05 rdma06 systemd[1]: Finished dnf makecache.
Apr  4 11:16:05 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.461s CPU time.
Apr  4 11:17:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 11:17:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 11:17:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 11:17:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.056 sec sweep
Apr  4 11:21:36 rdma06 chronyd[1034]: Selected source 10.18.100.10
Apr  4 11:22:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 11:22:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 11:22:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 11:22:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  4 11:27:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 11:27:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 11:27:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 11:27:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  4 11:32:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 11:32:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 11:32:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 11:32:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 11:37:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 11:37:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 11:37:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 11:37:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  4 11:42:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 11:42:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 11:42:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 11:42:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 11:47:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 11:47:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 11:47:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 11:47:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 11:52:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 11:52:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 11:52:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 11:52:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 11:57:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 11:57:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 11:57:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 11:57:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 12:02:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 12:02:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 12:02:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 12:02:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 12:07:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 12:07:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 12:07:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 12:07:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 12:12:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 12:12:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 12:12:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 12:12:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 12:17:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 12:17:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 12:17:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 12:17:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  4 12:22:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  4 12:22:01 rdma06 dnf[402861]: Updating Subscription Management repositories.
Apr  4 12:22:01 rdma06 dnf[402861]: Unable to read consumer identity
Apr  4 12:22:01 rdma06 dnf[402861]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  4 12:22:01 rdma06 dnf[402861]: Metadata cache refreshed recently.
Apr  4 12:22:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  4 12:22:01 rdma06 systemd[1]: Finished dnf makecache.
Apr  4 12:22:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 12:22:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 12:22:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 12:22:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  4 12:27:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 12:27:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 12:27:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 12:27:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  4 12:32:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 12:32:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 12:32:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 12:32:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  4 12:37:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 12:37:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 12:37:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 12:37:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 12:42:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 12:42:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 12:42:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 12:42:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 12:47:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 12:47:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 12:47:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 12:47:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 12:52:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 12:52:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 12:52:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 12:52:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 12:57:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 12:57:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 12:57:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 12:57:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 13:02:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 13:02:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 13:02:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 13:02:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  4 13:07:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 13:07:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 13:07:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 13:07:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  4 13:12:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 13:12:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 13:12:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 13:12:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 13:17:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 13:17:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 13:17:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 13:17:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 13:22:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 13:22:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 13:22:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 13:22:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  4 13:27:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 13:27:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 13:27:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 13:27:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 13:32:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 13:32:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 13:32:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 13:32:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 13:37:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 13:37:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 13:37:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 13:37:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 13:42:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 13:42:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 13:42:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 13:42:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 13:47:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 13:47:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 13:47:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 13:47:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 13:52:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 13:52:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 13:52:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 13:52:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 13:57:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 13:57:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 13:57:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 13:57:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 13:59:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  4 13:59:01 rdma06 dnf[406771]: Updating Subscription Management repositories.
Apr  4 13:59:01 rdma06 dnf[406771]: Unable to read consumer identity
Apr  4 13:59:01 rdma06 dnf[406771]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  4 13:59:01 rdma06 dnf[406771]: Metadata cache refreshed recently.
Apr  4 13:59:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  4 13:59:01 rdma06 systemd[1]: Finished dnf makecache.
Apr  4 14:02:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 14:02:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 14:02:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 14:02:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  4 14:07:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 14:07:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 14:07:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 14:07:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 14:12:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 14:12:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 14:12:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 14:12:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 14:17:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 14:17:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 14:17:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 14:17:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 14:22:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 14:22:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 14:22:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 14:22:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  4 14:27:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 14:27:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 14:27:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 14:27:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  4 14:32:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 14:32:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 14:32:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 14:32:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 14:37:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 14:37:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 14:37:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 14:37:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 14:42:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 14:42:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 14:42:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 14:42:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 14:47:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 14:47:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 14:47:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 14:47:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 14:52:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 14:52:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 14:52:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 14:52:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 14:57:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 14:57:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 14:57:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 14:57:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 15:02:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 15:02:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 15:02:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 15:02:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 15:07:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 15:07:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 15:07:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 15:07:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 15:12:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 15:12:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 15:12:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 15:12:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 15:17:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 15:17:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 15:17:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 15:17:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 15:20:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  4 15:20:01 rdma06 dnf[410053]: Updating Subscription Management repositories.
Apr  4 15:20:01 rdma06 dnf[410053]: Unable to read consumer identity
Apr  4 15:20:01 rdma06 dnf[410053]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  4 15:20:02 rdma06 dnf[410053]: Copr repo for fmf owned by psss                 3.2 kB/s | 3.3 kB     00:01
Apr  4 15:20:02 rdma06 dnf[410053]: beaker-AppStream                                 63 kB/s | 3.2 kB     00:00
Apr  4 15:20:02 rdma06 dnf[410053]: beaker-AppStream-debuginfo                       39 kB/s | 1.5 kB     00:00
Apr  4 15:20:02 rdma06 dnf[410053]: beaker-BaseOS                                    96 kB/s | 2.7 kB     00:00
Apr  4 15:20:02 rdma06 dnf[410053]: beaker-BaseOS-debuginfo                          36 kB/s | 1.5 kB     00:00
Apr  4 15:20:03 rdma06 dnf[410053]: beaker-CRB                                       47 kB/s | 2.7 kB     00:00
Apr  4 15:20:03 rdma06 dnf[410053]: beaker-CRB-debuginfo                             48 kB/s | 1.5 kB     00:00
Apr  4 15:20:03 rdma06 dnf[410053]: beaker-HighAvailability                          22 kB/s | 2.8 kB     00:00
Apr  4 15:20:03 rdma06 dnf[410053]: beaker-HighAvailability-debuginfo                45 kB/s | 1.5 kB     00:00
Apr  4 15:20:03 rdma06 dnf[410053]: beaker-NFV                                       52 kB/s | 2.7 kB     00:00
Apr  4 15:20:03 rdma06 dnf[410053]: beaker-NFV-debuginfo                             64 kB/s | 1.5 kB     00:00
Apr  4 15:20:03 rdma06 dnf[410053]: beaker-RT                                        61 kB/s | 2.7 kB     00:00
Apr  4 15:20:03 rdma06 dnf[410053]: beaker-RT-debuginfo                              28 kB/s | 1.5 kB     00:00
Apr  4 15:20:03 rdma06 dnf[410053]: beaker-ResilientStorage                          40 kB/s | 2.8 kB     00:00
Apr  4 15:20:03 rdma06 dnf[410053]: beaker-ResilientStorage-debuginfo                58 kB/s | 1.5 kB     00:00
Apr  4 15:20:03 rdma06 dnf[410053]: beaker-SAP                                      141 kB/s | 2.7 kB     00:00
Apr  4 15:20:03 rdma06 dnf[410053]: beaker-SAP-debuginfo                             28 kB/s | 1.5 kB     00:00
Apr  4 15:20:03 rdma06 dnf[410053]: beaker-SAPHANA                                   57 kB/s | 2.7 kB     00:00
Apr  4 15:20:03 rdma06 dnf[410053]: beaker-SAPHANA-debuginfo                         28 kB/s | 1.5 kB     00:00
Apr  4 15:20:04 rdma06 dnf[410053]: beaker-harness                                  1.1 kB/s | 1.3 kB     00:01
Apr  4 15:20:05 rdma06 dnf[410053]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr  4 15:20:05 rdma06 dnf[410053]: Metadata cache created.
Apr  4 15:20:05 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  4 15:20:05 rdma06 systemd[1]: Finished dnf makecache.
Apr  4 15:20:05 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.423s CPU time.
Apr  4 15:22:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 15:22:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 15:22:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 15:22:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  4 15:27:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 15:27:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 15:27:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 15:27:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 15:32:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 15:32:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 15:32:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 15:32:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 15:37:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 15:37:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 15:37:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 15:37:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 15:42:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 15:42:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 15:42:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 15:42:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 15:47:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 15:47:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 15:47:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 15:47:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 15:52:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 15:52:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 15:52:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 15:52:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 15:57:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 15:57:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 15:57:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 15:57:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 16:02:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 16:02:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 16:02:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 16:02:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 16:07:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 16:07:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 16:07:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 16:07:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  4 16:12:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 16:12:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 16:12:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 16:12:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 16:17:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 16:17:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 16:17:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 16:17:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 16:20:32 rdma06 NetworkManager[1025]: <info>  [1680639632.8913] dhcp4 (eno1): state changed new lease, address=10.73.131.45
Apr  4 16:22:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 16:22:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 16:22:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 16:22:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 16:27:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 16:27:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 16:27:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 16:27:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 16:32:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 16:32:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 16:32:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 16:32:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 16:34:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  4 16:34:01 rdma06 dnf[413064]: Updating Subscription Management repositories.
Apr  4 16:34:01 rdma06 dnf[413064]: Unable to read consumer identity
Apr  4 16:34:01 rdma06 dnf[413064]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  4 16:34:01 rdma06 dnf[413064]: Metadata cache refreshed recently.
Apr  4 16:34:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  4 16:34:01 rdma06 systemd[1]: Finished dnf makecache.
Apr  4 16:37:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 16:37:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 16:37:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 16:37:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  4 16:42:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 16:42:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 16:42:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 16:42:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 16:47:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 16:47:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 16:47:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 16:47:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 16:52:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 16:52:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 16:52:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 16:52:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 16:57:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 16:57:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 16:57:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 16:57:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 17:02:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 17:02:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 17:02:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 17:02:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 17:07:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 17:07:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 17:07:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 17:07:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 17:12:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 17:12:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 17:12:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 17:12:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 17:17:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 17:17:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 17:17:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 17:17:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 17:22:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 17:22:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 17:22:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 17:22:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 17:27:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 17:27:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 17:27:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 17:27:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 17:32:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 17:32:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 17:32:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 17:32:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 17:37:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 17:37:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 17:37:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 17:37:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 17:42:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 17:42:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 17:42:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 17:42:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  4 17:47:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 17:47:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 17:47:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 17:47:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  4 17:52:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 17:52:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 17:52:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 17:52:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 17:56:50 rdma06 systemd[1]: Starting dnf makecache...
Apr  4 17:56:51 rdma06 dnf[416411]: Updating Subscription Management repositories.
Apr  4 17:56:51 rdma06 dnf[416411]: Unable to read consumer identity
Apr  4 17:56:51 rdma06 dnf[416411]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  4 17:56:51 rdma06 dnf[416411]: Metadata cache refreshed recently.
Apr  4 17:56:51 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  4 17:56:51 rdma06 systemd[1]: Finished dnf makecache.
Apr  4 17:57:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 17:57:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 17:57:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 17:57:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  4 18:02:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 18:02:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 18:02:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 18:02:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 18:07:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 18:07:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 18:07:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 18:07:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 18:12:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 18:12:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 18:12:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 18:12:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 18:17:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 18:17:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 18:17:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 18:17:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  4 18:22:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 18:22:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 18:22:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 18:22:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  4 18:27:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 18:27:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 18:27:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 18:27:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 18:32:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 18:32:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 18:32:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 18:32:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  4 18:37:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 18:37:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 18:37:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 18:37:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 18:42:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 18:42:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 18:42:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 18:42:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 18:47:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 18:47:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 18:47:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 18:47:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 18:52:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 18:52:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 18:52:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 18:52:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  4 18:57:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 18:57:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 18:57:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 18:57:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 19:02:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 19:02:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 19:02:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 19:02:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 19:07:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 19:07:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 19:07:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 19:07:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 19:12:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 19:12:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 19:12:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 19:12:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 19:17:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 19:17:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 19:17:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 19:17:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  4 19:22:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 19:22:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 19:22:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 19:22:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 19:27:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 19:27:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 19:27:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 19:27:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  4 19:32:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 19:32:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 19:32:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 19:32:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 19:37:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 19:37:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 19:37:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 19:37:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 19:42:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 19:42:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 19:42:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 19:42:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 19:47:20 rdma06 systemd[1]: Starting dnf makecache...
Apr  4 19:47:21 rdma06 dnf[420882]: Updating Subscription Management repositories.
Apr  4 19:47:21 rdma06 dnf[420882]: Unable to read consumer identity
Apr  4 19:47:21 rdma06 dnf[420882]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  4 19:47:22 rdma06 dnf[420882]: Copr repo for fmf owned by psss                 3.1 kB/s | 3.3 kB     00:01
Apr  4 19:47:22 rdma06 dnf[420882]: beaker-AppStream                                 51 kB/s | 3.2 kB     00:00
Apr  4 19:47:22 rdma06 dnf[420882]: beaker-AppStream-debuginfo                       38 kB/s | 1.5 kB     00:00
Apr  4 19:47:22 rdma06 dnf[420882]: beaker-BaseOS                                    62 kB/s | 2.7 kB     00:00
Apr  4 19:47:23 rdma06 dnf[420882]: beaker-BaseOS-debuginfo                          39 kB/s | 1.5 kB     00:00
Apr  4 19:47:23 rdma06 dnf[420882]: beaker-CRB                                       67 kB/s | 2.7 kB     00:00
Apr  4 19:47:23 rdma06 dnf[420882]: beaker-CRB-debuginfo                             85 kB/s | 1.5 kB     00:00
Apr  4 19:47:23 rdma06 dnf[420882]: beaker-HighAvailability                          83 kB/s | 2.8 kB     00:00
Apr  4 19:47:23 rdma06 dnf[420882]: beaker-HighAvailability-debuginfo                40 kB/s | 1.5 kB     00:00
Apr  4 19:47:23 rdma06 dnf[420882]: beaker-NFV                                       54 kB/s | 2.7 kB     00:00
Apr  4 19:47:23 rdma06 dnf[420882]: beaker-NFV-debuginfo                             62 kB/s | 1.5 kB     00:00
Apr  4 19:47:23 rdma06 dnf[420882]: beaker-RT                                        67 kB/s | 2.7 kB     00:00
Apr  4 19:47:23 rdma06 dnf[420882]: beaker-RT-debuginfo                              92 kB/s | 1.5 kB     00:00
Apr  4 19:47:23 rdma06 dnf[420882]: beaker-ResilientStorage                          58 kB/s | 2.8 kB     00:00
Apr  4 19:47:23 rdma06 dnf[420882]: beaker-ResilientStorage-debuginfo                62 kB/s | 1.5 kB     00:00
Apr  4 19:47:23 rdma06 dnf[420882]: beaker-SAP                                       56 kB/s | 2.7 kB     00:00
Apr  4 19:47:23 rdma06 dnf[420882]: beaker-SAP-debuginfo                             49 kB/s | 1.5 kB     00:00
Apr  4 19:47:23 rdma06 dnf[420882]: beaker-SAPHANA                                   83 kB/s | 2.7 kB     00:00
Apr  4 19:47:23 rdma06 dnf[420882]: beaker-SAPHANA-debuginfo                         26 kB/s | 1.5 kB     00:00
Apr  4 19:47:24 rdma06 dnf[420882]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr  4 19:47:24 rdma06 dnf[420882]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr  4 19:47:25 rdma06 dnf[420882]: Metadata cache created.
Apr  4 19:47:25 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  4 19:47:25 rdma06 systemd[1]: Finished dnf makecache.
Apr  4 19:47:25 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.410s CPU time.
Apr  4 19:47:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 19:47:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 19:47:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 19:47:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  4 19:52:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 19:52:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 19:52:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 19:52:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  4 19:57:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 19:57:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 19:57:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 19:57:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 20:02:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 20:02:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 20:02:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 20:02:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  4 20:07:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 20:07:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 20:07:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 20:07:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 20:12:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 20:12:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 20:12:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 20:12:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 20:17:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 20:17:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 20:17:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 20:17:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 20:22:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 20:22:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 20:22:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 20:22:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 20:27:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 20:27:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 20:27:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 20:27:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 20:32:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 20:32:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 20:32:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 20:32:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 20:33:27 rdma06 chronyd[1034]: Selected source 10.13.199.1
Apr  4 20:37:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 20:37:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 20:37:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 20:37:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 20:42:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 20:42:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 20:42:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 20:42:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 20:47:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 20:47:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 20:47:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 20:47:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 20:52:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 20:52:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 20:52:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 20:52:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 20:57:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 20:57:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 20:57:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 20:57:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 21:02:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 21:02:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 21:02:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 21:02:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 21:07:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 21:07:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 21:07:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 21:07:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 21:12:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 21:12:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 21:12:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 21:12:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 21:17:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 21:17:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 21:17:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 21:17:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 21:22:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 21:22:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 21:22:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 21:22:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.056 sec sweep
Apr  4 21:27:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 21:27:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 21:27:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 21:27:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 21:32:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 21:32:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 21:32:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 21:32:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 21:37:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 21:37:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 21:37:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 21:37:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 21:41:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  4 21:41:01 rdma06 dnf[425499]: Updating Subscription Management repositories.
Apr  4 21:41:01 rdma06 dnf[425499]: Unable to read consumer identity
Apr  4 21:41:01 rdma06 dnf[425499]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  4 21:41:01 rdma06 dnf[425499]: Metadata cache refreshed recently.
Apr  4 21:41:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  4 21:41:01 rdma06 systemd[1]: Finished dnf makecache.
Apr  4 21:42:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 21:42:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 21:42:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 21:42:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  4 21:47:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 21:47:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 21:47:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 21:47:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 21:52:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 21:52:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 21:52:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 21:52:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 21:57:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 21:57:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 21:57:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 21:57:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 22:02:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 22:02:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 22:02:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 22:02:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 22:07:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 22:07:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 22:07:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 22:07:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 22:12:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 22:12:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 22:12:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 22:12:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 22:17:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 22:17:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 22:17:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 22:17:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 22:20:32 rdma06 NetworkManager[1025]: <info>  [1680661232.9348] dhcp4 (eno1): state changed new lease, address=10.73.131.45
Apr  4 22:22:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 22:22:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 22:22:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 22:22:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  4 22:27:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 22:27:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 22:27:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 22:27:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  4 22:32:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 22:32:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 22:32:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 22:32:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 22:33:00 rdma06 systemd[5685]: Starting Cleanup of User's Temporary Files and Directories...
Apr  4 22:33:00 rdma06 systemd[5685]: Finished Cleanup of User's Temporary Files and Directories.
Apr  4 22:37:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 22:37:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 22:37:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 22:37:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 22:40:00 rdma06 systemd[1]: Starting Cleanup of Temporary Directories...
Apr  4 22:40:00 rdma06 systemd[1]: systemd-tmpfiles-clean.service: Deactivated successfully.
Apr  4 22:40:00 rdma06 systemd[1]: Finished Cleanup of Temporary Directories.
Apr  4 22:40:00 rdma06 systemd[1]: run-credentials-systemd\x2dtmpfiles\x2dclean.service.mount: Deactivated successfully.
Apr  4 22:42:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 22:42:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 22:42:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 22:42:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  4 22:47:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 22:47:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 22:47:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 22:47:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 22:52:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 22:52:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 22:52:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 22:52:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  4 22:57:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 22:57:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 22:57:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 22:57:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 23:02:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 23:02:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 23:02:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 23:02:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 23:07:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 23:07:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 23:07:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 23:07:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 23:12:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 23:12:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 23:12:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 23:12:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 23:14:50 rdma06 systemd[1]: Starting dnf makecache...
Apr  4 23:14:51 rdma06 dnf[429309]: Updating Subscription Management repositories.
Apr  4 23:14:51 rdma06 dnf[429309]: Unable to read consumer identity
Apr  4 23:14:51 rdma06 dnf[429309]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  4 23:14:53 rdma06 dnf[429309]: Copr repo for fmf owned by psss                 2.1 kB/s | 3.3 kB     00:01
Apr  4 23:14:53 rdma06 dnf[429309]: beaker-AppStream                                 56 kB/s | 3.2 kB     00:00
Apr  4 23:14:53 rdma06 dnf[429309]: beaker-AppStream-debuginfo                       56 kB/s | 1.5 kB     00:00
Apr  4 23:14:53 rdma06 dnf[429309]: beaker-BaseOS                                   175 kB/s | 2.7 kB     00:00
Apr  4 23:14:53 rdma06 dnf[429309]: beaker-BaseOS-debuginfo                          36 kB/s | 1.5 kB     00:00
Apr  4 23:14:53 rdma06 dnf[429309]: beaker-CRB                                       33 kB/s | 2.7 kB     00:00
Apr  4 23:14:53 rdma06 dnf[429309]: beaker-CRB-debuginfo                             23 kB/s | 1.5 kB     00:00
Apr  4 23:14:53 rdma06 dnf[429309]: beaker-HighAvailability                          36 kB/s | 2.8 kB     00:00
Apr  4 23:14:53 rdma06 dnf[429309]: beaker-HighAvailability-debuginfo                40 kB/s | 1.5 kB     00:00
Apr  4 23:14:53 rdma06 dnf[429309]: beaker-NFV                                      101 kB/s | 2.7 kB     00:00
Apr  4 23:14:53 rdma06 dnf[429309]: beaker-NFV-debuginfo                             59 kB/s | 1.5 kB     00:00
Apr  4 23:14:53 rdma06 dnf[429309]: beaker-RT                                        57 kB/s | 2.7 kB     00:00
Apr  4 23:14:54 rdma06 dnf[429309]: beaker-RT-debuginfo                              70 kB/s | 1.5 kB     00:00
Apr  4 23:14:54 rdma06 dnf[429309]: beaker-ResilientStorage                          59 kB/s | 2.8 kB     00:00
Apr  4 23:14:54 rdma06 dnf[429309]: beaker-ResilientStorage-debuginfo                57 kB/s | 1.5 kB     00:00
Apr  4 23:14:54 rdma06 dnf[429309]: beaker-SAP                                      147 kB/s | 2.7 kB     00:00
Apr  4 23:14:54 rdma06 dnf[429309]: beaker-SAP-debuginfo                             58 kB/s | 1.5 kB     00:00
Apr  4 23:14:54 rdma06 dnf[429309]: beaker-SAPHANA                                   59 kB/s | 2.7 kB     00:00
Apr  4 23:14:54 rdma06 dnf[429309]: beaker-SAPHANA-debuginfo                         50 kB/s | 1.5 kB     00:00
Apr  4 23:14:54 rdma06 dnf[429309]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr  4 23:14:55 rdma06 dnf[429309]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr  4 23:14:55 rdma06 dnf[429309]: Metadata cache created.
Apr  4 23:14:55 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  4 23:14:55 rdma06 systemd[1]: Finished dnf makecache.
Apr  4 23:14:55 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.486s CPU time.
Apr  4 23:17:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 23:17:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 23:17:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 23:17:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.056 sec sweep
Apr  4 23:22:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 23:22:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 23:22:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 23:22:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 23:27:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 23:27:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 23:27:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 23:27:33 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 23:32:33 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 23:32:33 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 23:32:33 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 23:32:33 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 23:37:33 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 23:37:33 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 23:37:33 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 23:37:33 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 23:42:33 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 23:42:33 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 23:42:33 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 23:42:33 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 23:47:33 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 23:47:33 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 23:47:33 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 23:47:33 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 23:52:33 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 23:52:33 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 23:52:33 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 23:52:33 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  4 23:57:33 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  4 23:57:33 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  4 23:57:33 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  4 23:57:33 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 00:01:00 rdma06 systemd[1]: Starting Rotate log files...
Apr  5 00:01:00 rdma06 systemd[1]: logrotate.service: Deactivated successfully.
Apr  5 00:01:00 rdma06 systemd[1]: Finished Rotate log files.
Apr  5 00:02:33 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 00:02:33 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 00:02:33 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 00:02:33 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  5 00:07:33 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 00:07:33 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 00:07:33 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 00:07:33 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  5 00:12:33 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 00:12:33 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 00:12:33 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 00:12:33 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 00:17:33 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 00:17:33 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 00:17:33 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 00:17:33 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 00:22:34 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 00:22:34 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 00:22:34 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 00:22:34 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 00:25:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  5 00:25:01 rdma06 dnf[432177]: Updating Subscription Management repositories.
Apr  5 00:25:01 rdma06 dnf[432177]: Unable to read consumer identity
Apr  5 00:25:01 rdma06 dnf[432177]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  5 00:25:01 rdma06 dnf[432177]: Metadata cache refreshed recently.
Apr  5 00:25:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  5 00:25:01 rdma06 systemd[1]: Finished dnf makecache.
Apr  5 00:27:34 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 00:27:34 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 00:27:34 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 00:27:34 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  5 00:32:34 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 00:32:34 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 00:32:34 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 00:32:34 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 00:37:34 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 00:37:34 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 00:37:34 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 00:37:34 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 00:42:34 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 00:42:34 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 00:42:34 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 00:42:34 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 00:47:34 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 00:47:34 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 00:47:34 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 00:47:34 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  5 00:52:34 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 00:52:34 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 00:52:34 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 00:52:34 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 00:57:34 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 00:57:34 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 00:57:34 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 00:57:34 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 01:02:34 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 01:02:34 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 01:02:34 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 01:02:34 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  5 01:07:34 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 01:07:34 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 01:07:34 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 01:07:35 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  5 01:12:35 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 01:12:35 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 01:12:35 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 01:12:35 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 01:17:35 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 01:17:35 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 01:17:35 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 01:17:35 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 01:21:45 rdma06 systemd[1]: session-8.scope: Deactivated successfully.
Apr  5 01:21:45 rdma06 systemd[1]: session-8.scope: Consumed 47.141s CPU time.
Apr  5 01:21:45 rdma06 systemd-logind[1064]: Session 8 logged out. Waiting for processes to exit.
Apr  5 01:21:45 rdma06 systemd-logind[1064]: Removed session 8.
Apr  5 01:22:35 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 01:22:35 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 01:22:35 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 01:22:35 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 01:27:35 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 01:27:35 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 01:27:35 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 01:27:35 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 01:32:35 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 01:32:35 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 01:32:35 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 01:32:35 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 01:37:35 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 01:37:35 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 01:37:35 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 01:37:35 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 01:42:35 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 01:42:35 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 01:42:35 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 01:42:35 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 01:47:35 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 01:47:35 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 01:47:35 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 01:47:35 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 01:52:35 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 01:52:35 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 01:52:35 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 01:52:35 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 01:57:35 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 01:57:35 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 01:57:35 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 01:57:36 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 02:02:36 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 02:02:36 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 02:02:36 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 02:02:36 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 02:07:36 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 02:07:36 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 02:07:36 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 02:07:36 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 02:12:36 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 02:12:36 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 02:12:36 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 02:12:36 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 02:16:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  5 02:16:01 rdma06 dnf[436678]: Updating Subscription Management repositories.
Apr  5 02:16:01 rdma06 dnf[436678]: Unable to read consumer identity
Apr  5 02:16:01 rdma06 dnf[436678]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  5 02:16:02 rdma06 dnf[436678]: Copr repo for fmf owned by psss                 2.6 kB/s | 3.3 kB     00:01
Apr  5 02:16:02 rdma06 dnf[436678]: beaker-AppStream                                 50 kB/s | 3.2 kB     00:00
Apr  5 02:16:02 rdma06 dnf[436678]: beaker-AppStream-debuginfo                       24 kB/s | 1.5 kB     00:00
Apr  5 02:16:03 rdma06 dnf[436678]: beaker-BaseOS                                   151 kB/s | 2.7 kB     00:00
Apr  5 02:16:03 rdma06 dnf[436678]: beaker-BaseOS-debuginfo                          33 kB/s | 1.5 kB     00:00
Apr  5 02:16:03 rdma06 dnf[436678]: beaker-CRB                                       51 kB/s | 2.7 kB     00:00
Apr  5 02:16:03 rdma06 dnf[436678]: beaker-CRB-debuginfo                             76 kB/s | 1.5 kB     00:00
Apr  5 02:16:03 rdma06 dnf[436678]: beaker-HighAvailability                          91 kB/s | 2.8 kB     00:00
Apr  5 02:16:03 rdma06 dnf[436678]: beaker-HighAvailability-debuginfo                51 kB/s | 1.5 kB     00:00
Apr  5 02:16:03 rdma06 dnf[436678]: beaker-NFV                                       79 kB/s | 2.7 kB     00:00
Apr  5 02:16:03 rdma06 dnf[436678]: beaker-NFV-debuginfo                             64 kB/s | 1.5 kB     00:00
Apr  5 02:16:03 rdma06 dnf[436678]: beaker-RT                                        67 kB/s | 2.7 kB     00:00
Apr  5 02:16:03 rdma06 dnf[436678]: beaker-RT-debuginfo                              35 kB/s | 1.5 kB     00:00
Apr  5 02:16:03 rdma06 dnf[436678]: beaker-ResilientStorage                          60 kB/s | 2.8 kB     00:00
Apr  5 02:16:03 rdma06 dnf[436678]: beaker-ResilientStorage-debuginfo                63 kB/s | 1.5 kB     00:00
Apr  5 02:16:03 rdma06 dnf[436678]: beaker-SAP                                       90 kB/s | 2.7 kB     00:00
Apr  5 02:16:03 rdma06 dnf[436678]: beaker-SAP-debuginfo                             48 kB/s | 1.5 kB     00:00
Apr  5 02:16:03 rdma06 dnf[436678]: beaker-SAPHANA                                   54 kB/s | 2.7 kB     00:00
Apr  5 02:16:03 rdma06 dnf[436678]: beaker-SAPHANA-debuginfo                         44 kB/s | 1.5 kB     00:00
Apr  5 02:16:04 rdma06 dnf[436678]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr  5 02:16:05 rdma06 dnf[436678]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr  5 02:16:05 rdma06 dnf[436678]: Metadata cache created.
Apr  5 02:16:05 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  5 02:16:05 rdma06 systemd[1]: Finished dnf makecache.
Apr  5 02:16:05 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.354s CPU time.
Apr  5 02:17:36 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 02:17:36 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 02:17:36 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 02:17:36 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 02:22:36 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 02:22:36 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 02:22:36 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 02:22:36 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.056 sec sweep
Apr  5 02:27:36 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 02:27:36 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 02:27:36 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 02:27:36 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 02:32:36 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 02:32:36 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 02:32:36 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 02:32:36 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 02:37:36 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 02:37:36 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 02:37:36 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 02:37:36 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 02:42:36 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 02:42:36 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 02:42:36 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 02:42:36 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 02:47:36 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 02:47:36 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 02:47:36 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 02:47:37 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 02:52:37 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 02:52:37 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 02:52:37 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 02:52:37 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 02:57:37 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 02:57:37 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 02:57:37 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 02:57:37 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 03:02:37 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 03:02:37 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 03:02:37 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 03:02:37 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 03:07:37 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 03:07:37 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 03:07:37 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 03:07:37 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  5 03:12:37 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 03:12:37 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 03:12:37 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 03:12:37 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 03:17:37 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 03:17:37 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 03:17:37 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 03:17:37 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 03:22:37 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 03:22:37 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 03:22:37 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 03:22:37 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 03:27:37 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 03:27:37 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 03:27:37 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 03:27:37 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 03:32:37 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 03:32:37 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 03:32:37 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 03:32:37 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 03:37:37 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 03:37:37 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 03:37:37 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 03:37:37 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 03:42:37 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 03:42:37 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 03:42:37 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 03:42:38 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 03:47:38 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 03:47:38 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 03:47:38 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 03:47:38 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 03:52:38 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 03:52:38 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 03:52:38 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 03:52:38 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 03:57:38 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 03:57:38 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 03:57:38 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 03:57:38 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 03:58:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  5 03:58:01 rdma06 dnf[440818]: Updating Subscription Management repositories.
Apr  5 03:58:01 rdma06 dnf[440818]: Unable to read consumer identity
Apr  5 03:58:01 rdma06 dnf[440818]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  5 03:58:01 rdma06 dnf[440818]: Metadata cache refreshed recently.
Apr  5 03:58:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  5 03:58:01 rdma06 systemd[1]: Finished dnf makecache.
Apr  5 04:02:38 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 04:02:38 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 04:02:38 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 04:02:38 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  5 04:07:38 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 04:07:38 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 04:07:38 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 04:07:38 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 04:12:38 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 04:12:38 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 04:12:38 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 04:12:38 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 04:17:38 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 04:17:38 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 04:17:38 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 04:17:38 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 04:20:32 rdma06 NetworkManager[1025]: <info>  [1680682832.8715] dhcp4 (eno1): state changed new lease, address=10.73.131.45
Apr  5 04:22:38 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 04:22:38 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 04:22:38 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 04:22:38 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 04:27:38 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 04:27:38 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 04:27:38 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 04:27:38 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 04:32:38 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 04:32:38 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 04:32:38 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 04:32:38 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 04:37:38 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 04:37:38 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 04:37:38 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 04:37:39 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 04:42:39 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 04:42:39 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 04:42:39 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 04:42:39 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 04:47:39 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 04:47:39 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 04:47:39 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 04:47:39 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 04:52:39 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 04:52:39 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 04:52:39 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 04:52:39 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 04:57:39 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 04:57:39 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 04:57:39 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 04:57:39 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 05:02:39 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 05:02:39 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 05:02:39 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 05:02:39 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 05:07:39 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 05:07:39 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 05:07:39 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 05:07:39 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 05:12:39 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 05:12:39 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 05:12:39 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 05:12:39 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 05:17:39 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 05:17:39 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 05:17:39 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 05:17:39 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 05:22:39 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 05:22:39 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 05:22:39 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 05:22:39 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 05:27:39 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 05:27:39 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 05:27:39 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 05:27:40 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 05:29:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  5 05:29:01 rdma06 dnf[444503]: Updating Subscription Management repositories.
Apr  5 05:29:01 rdma06 dnf[444503]: Unable to read consumer identity
Apr  5 05:29:01 rdma06 dnf[444503]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  5 05:29:02 rdma06 dnf[444503]: Copr repo for fmf owned by psss                 2.7 kB/s | 3.3 kB     00:01
Apr  5 05:29:02 rdma06 dnf[444503]: beaker-AppStream                                125 kB/s | 3.2 kB     00:00
Apr  5 05:29:02 rdma06 dnf[444503]: beaker-AppStream-debuginfo                      130 kB/s | 1.5 kB     00:00
Apr  5 05:29:03 rdma06 dnf[444503]: beaker-BaseOS                                   155 kB/s | 2.7 kB     00:00
Apr  5 05:29:03 rdma06 dnf[444503]: beaker-BaseOS-debuginfo                         132 kB/s | 1.5 kB     00:00
Apr  5 05:29:03 rdma06 dnf[444503]: beaker-CRB                                      191 kB/s | 2.7 kB     00:00
Apr  5 05:29:03 rdma06 dnf[444503]: beaker-CRB-debuginfo                            135 kB/s | 1.5 kB     00:00
Apr  5 05:29:03 rdma06 dnf[444503]: beaker-HighAvailability                         207 kB/s | 2.8 kB     00:00
Apr  5 05:29:03 rdma06 dnf[444503]: beaker-HighAvailability-debuginfo               167 kB/s | 1.5 kB     00:00
Apr  5 05:29:03 rdma06 dnf[444503]: beaker-NFV                                      377 kB/s | 2.7 kB     00:00
Apr  5 05:29:03 rdma06 dnf[444503]: beaker-NFV-debuginfo                            105 kB/s | 1.5 kB     00:00
Apr  5 05:29:03 rdma06 dnf[444503]: beaker-RT                                       290 kB/s | 2.7 kB     00:00
Apr  5 05:29:03 rdma06 dnf[444503]: beaker-RT-debuginfo                             305 kB/s | 1.5 kB     00:00
Apr  5 05:29:03 rdma06 dnf[444503]: beaker-ResilientStorage                         126 kB/s | 2.8 kB     00:00
Apr  5 05:29:03 rdma06 dnf[444503]: beaker-ResilientStorage-debuginfo               127 kB/s | 1.5 kB     00:00
Apr  5 05:29:03 rdma06 dnf[444503]: beaker-SAP                                      307 kB/s | 2.7 kB     00:00
Apr  5 05:29:03 rdma06 dnf[444503]: beaker-SAP-debuginfo                            104 kB/s | 1.5 kB     00:00
Apr  5 05:29:03 rdma06 dnf[444503]: beaker-SAPHANA                                  581 kB/s | 2.7 kB     00:00
Apr  5 05:29:03 rdma06 dnf[444503]: beaker-SAPHANA-debuginfo                        126 kB/s | 1.5 kB     00:00
Apr  5 05:29:04 rdma06 dnf[444503]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr  5 05:29:04 rdma06 dnf[444503]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr  5 05:29:05 rdma06 dnf[444503]: Metadata cache created.
Apr  5 05:29:05 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  5 05:29:05 rdma06 systemd[1]: Finished dnf makecache.
Apr  5 05:29:05 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.415s CPU time.
Apr  5 05:32:40 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 05:32:40 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 05:32:40 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 05:32:40 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  5 05:37:40 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 05:37:40 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 05:37:40 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 05:37:40 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 05:42:40 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 05:42:40 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 05:42:40 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 05:42:40 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 05:47:40 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 05:47:40 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 05:47:40 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 05:47:40 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 05:52:40 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 05:52:40 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 05:52:40 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 05:52:40 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 05:57:40 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 05:57:40 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 05:57:40 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 05:57:40 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 06:02:40 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 06:02:40 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 06:02:40 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 06:02:40 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 06:07:40 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 06:07:40 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 06:07:40 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 06:07:40 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 06:12:40 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 06:12:40 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 06:12:40 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 06:12:40 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 06:17:40 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 06:17:40 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 06:17:40 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 06:17:40 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 06:22:41 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 06:22:41 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 06:22:41 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 06:22:41 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  5 06:27:41 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 06:27:41 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 06:27:41 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 06:27:41 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 06:32:41 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 06:32:41 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 06:32:41 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 06:32:41 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 06:37:41 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 06:37:41 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 06:37:41 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 06:37:41 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 06:42:41 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 06:42:41 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 06:42:41 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 06:42:41 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 06:47:41 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 06:47:41 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 06:47:41 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 06:47:41 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 06:52:41 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 06:52:41 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 06:52:41 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 06:52:41 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 06:57:41 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 06:57:41 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 06:57:41 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 06:57:41 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 07:02:41 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 07:02:41 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 07:02:41 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 07:02:41 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 07:07:41 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 07:07:41 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 07:07:41 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 07:07:41 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 07:12:42 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 07:12:42 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 07:12:42 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 07:12:42 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 07:14:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  5 07:14:01 rdma06 dnf[448773]: Updating Subscription Management repositories.
Apr  5 07:14:01 rdma06 dnf[448773]: Unable to read consumer identity
Apr  5 07:14:01 rdma06 dnf[448773]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  5 07:14:01 rdma06 dnf[448773]: Metadata cache refreshed recently.
Apr  5 07:14:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  5 07:14:01 rdma06 systemd[1]: Finished dnf makecache.
Apr  5 07:17:42 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 07:17:42 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 07:17:42 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 07:17:42 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  5 07:22:42 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 07:22:42 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 07:22:42 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 07:22:42 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 07:27:42 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 07:27:42 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 07:27:42 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 07:27:42 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 07:32:42 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 07:32:42 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 07:32:42 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 07:32:42 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 07:37:42 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 07:37:42 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 07:37:42 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 07:37:42 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 07:42:42 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 07:42:42 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 07:42:42 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 07:42:42 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 07:47:42 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 07:47:42 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 07:47:42 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 07:47:42 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  5 07:52:42 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 07:52:42 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 07:52:42 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 07:52:42 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 07:57:42 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 07:57:42 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 07:57:42 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 07:57:42 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 08:02:42 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 08:02:42 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 08:02:42 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 08:02:43 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 08:07:43 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 08:07:43 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 08:07:43 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 08:07:43 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 08:12:43 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 08:12:43 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 08:12:43 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 08:12:43 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 08:17:43 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 08:17:43 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 08:17:43 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 08:17:43 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 08:22:43 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 08:22:43 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 08:22:43 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 08:22:43 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 08:27:43 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 08:27:43 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 08:27:43 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 08:27:43 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 08:32:43 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 08:32:43 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 08:32:43 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 08:32:43 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 08:37:43 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 08:37:43 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 08:37:43 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 08:37:43 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 08:42:43 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 08:42:43 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 08:42:43 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 08:42:43 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 08:47:43 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 08:47:43 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 08:47:43 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 08:47:43 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 08:52:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  5 08:52:01 rdma06 dnf[452728]: Updating Subscription Management repositories.
Apr  5 08:52:01 rdma06 dnf[452728]: Unable to read consumer identity
Apr  5 08:52:01 rdma06 dnf[452728]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  5 08:52:02 rdma06 dnf[452728]: Copr repo for fmf owned by psss                 3.5 kB/s | 3.3 kB     00:00
Apr  5 08:52:02 rdma06 dnf[452728]: beaker-AppStream                                208 kB/s | 3.2 kB     00:00
Apr  5 08:52:02 rdma06 dnf[452728]: beaker-AppStream-debuginfo                      107 kB/s | 1.5 kB     00:00
Apr  5 08:52:02 rdma06 dnf[452728]: beaker-BaseOS                                   347 kB/s | 2.7 kB     00:00
Apr  5 08:52:02 rdma06 dnf[452728]: beaker-BaseOS-debuginfo                         122 kB/s | 1.5 kB     00:00
Apr  5 08:52:02 rdma06 dnf[452728]: beaker-CRB                                      272 kB/s | 2.7 kB     00:00
Apr  5 08:52:02 rdma06 dnf[452728]: beaker-CRB-debuginfo                            117 kB/s | 1.5 kB     00:00
Apr  5 08:52:02 rdma06 dnf[452728]: beaker-HighAvailability                         228 kB/s | 2.8 kB     00:00
Apr  5 08:52:02 rdma06 dnf[452728]: beaker-HighAvailability-debuginfo                68 kB/s | 1.5 kB     00:00
Apr  5 08:52:02 rdma06 dnf[452728]: beaker-NFV                                      194 kB/s | 2.7 kB     00:00
Apr  5 08:52:02 rdma06 dnf[452728]: beaker-NFV-debuginfo                            111 kB/s | 1.5 kB     00:00
Apr  5 08:52:02 rdma06 dnf[452728]: beaker-RT                                       157 kB/s | 2.7 kB     00:00
Apr  5 08:52:03 rdma06 dnf[452728]: beaker-RT-debuginfo                             295 kB/s | 1.5 kB     00:00
Apr  5 08:52:03 rdma06 dnf[452728]: beaker-ResilientStorage                         152 kB/s | 2.8 kB     00:00
Apr  5 08:52:03 rdma06 dnf[452728]: beaker-ResilientStorage-debuginfo               132 kB/s | 1.5 kB     00:00
Apr  5 08:52:03 rdma06 dnf[452728]: beaker-SAP                                      262 kB/s | 2.7 kB     00:00
Apr  5 08:52:03 rdma06 dnf[452728]: beaker-SAP-debuginfo                            107 kB/s | 1.5 kB     00:00
Apr  5 08:52:03 rdma06 dnf[452728]: beaker-SAPHANA                                  495 kB/s | 2.7 kB     00:00
Apr  5 08:52:03 rdma06 dnf[452728]: beaker-SAPHANA-debuginfo                        112 kB/s | 1.5 kB     00:00
Apr  5 08:52:03 rdma06 dnf[452728]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr  5 08:52:04 rdma06 dnf[452728]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr  5 08:52:04 rdma06 dnf[452728]: Metadata cache created.
Apr  5 08:52:04 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  5 08:52:04 rdma06 systemd[1]: Finished dnf makecache.
Apr  5 08:52:04 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.441s CPU time.
Apr  5 08:52:43 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 08:52:43 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 08:52:43 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 08:52:44 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  5 08:57:44 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 08:57:44 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 08:57:44 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 08:57:44 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 09:02:44 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 09:02:44 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 09:02:44 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 09:02:44 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 09:07:44 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 09:07:44 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 09:07:44 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 09:07:44 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 09:12:44 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 09:12:44 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 09:12:44 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 09:12:44 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 09:17:44 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 09:17:44 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 09:17:44 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 09:17:44 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 09:22:44 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 09:22:44 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 09:22:44 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 09:22:44 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 09:27:44 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 09:27:44 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 09:27:44 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 09:27:44 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 09:32:44 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 09:32:44 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 09:32:44 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 09:32:44 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 09:37:44 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 09:37:44 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 09:37:44 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 09:37:44 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 09:42:44 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 09:42:44 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 09:42:44 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 09:42:44 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  5 09:47:45 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 09:47:45 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 09:47:45 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 09:47:45 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 09:52:45 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 09:52:45 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 09:52:45 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 09:52:45 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 09:57:45 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 09:57:45 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 09:57:45 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 09:57:45 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 10:02:45 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 10:02:45 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 10:02:45 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 10:02:45 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 10:07:45 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 10:07:45 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 10:07:45 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 10:07:45 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 10:12:45 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 10:12:45 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 10:12:45 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 10:12:45 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 10:17:45 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 10:17:45 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 10:17:45 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 10:17:45 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 10:20:32 rdma06 NetworkManager[1025]: <info>  [1680704432.9229] dhcp4 (eno1): state changed new lease, address=10.73.131.45
Apr  5 10:22:45 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 10:22:45 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 10:22:45 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 10:22:45 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  5 10:27:45 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 10:27:45 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 10:27:45 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 10:27:45 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 10:32:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  5 10:32:01 rdma06 dnf[456792]: Updating Subscription Management repositories.
Apr  5 10:32:01 rdma06 dnf[456792]: Unable to read consumer identity
Apr  5 10:32:01 rdma06 dnf[456792]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  5 10:32:01 rdma06 dnf[456792]: Metadata cache refreshed recently.
Apr  5 10:32:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  5 10:32:01 rdma06 systemd[1]: Finished dnf makecache.
Apr  5 10:32:45 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 10:32:45 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 10:32:45 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 10:32:45 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 10:37:46 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 10:37:46 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 10:37:46 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 10:37:46 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 10:42:46 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 10:42:46 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 10:42:46 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 10:42:46 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 10:47:46 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 10:47:46 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 10:47:46 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 10:47:46 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 10:52:46 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 10:52:46 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 10:52:46 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 10:52:46 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 10:57:46 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 10:57:46 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 10:57:46 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 10:57:46 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 11:02:46 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 11:02:46 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 11:02:46 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 11:02:46 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 11:07:46 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 11:07:46 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 11:07:46 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 11:07:46 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 11:12:46 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 11:12:46 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 11:12:46 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 11:12:46 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 11:17:46 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 11:17:46 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 11:17:46 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 11:17:46 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 11:22:46 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 11:22:46 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 11:22:46 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 11:22:46 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 11:27:47 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 11:27:47 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 11:27:47 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 11:27:47 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 11:32:47 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 11:32:47 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 11:32:47 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 11:32:47 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 11:37:47 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 11:37:47 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 11:37:47 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 11:37:47 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 11:41:50 rdma06 systemd[1]: Starting dnf makecache...
Apr  5 11:41:51 rdma06 dnf[459618]: Updating Subscription Management repositories.
Apr  5 11:41:51 rdma06 dnf[459618]: Unable to read consumer identity
Apr  5 11:41:51 rdma06 dnf[459618]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  5 11:41:51 rdma06 dnf[459618]: Metadata cache refreshed recently.
Apr  5 11:41:51 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  5 11:41:51 rdma06 systemd[1]: Finished dnf makecache.
Apr  5 11:42:47 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 11:42:47 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 11:42:47 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 11:42:47 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 11:47:47 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 11:47:47 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 11:47:47 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 11:47:47 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  5 11:52:47 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 11:52:47 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 11:52:47 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 11:52:47 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 11:57:47 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 11:57:47 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 11:57:47 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 11:57:47 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 12:02:47 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 12:02:47 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 12:02:47 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 12:02:47 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 12:07:47 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 12:07:47 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 12:07:47 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 12:07:47 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 12:12:47 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 12:12:47 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 12:12:47 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 12:12:47 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 12:17:48 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 12:17:48 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 12:17:48 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 12:17:48 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 12:22:48 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 12:22:48 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 12:22:48 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 12:22:48 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 12:27:48 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 12:27:48 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 12:27:48 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 12:27:48 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 12:32:48 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 12:32:48 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 12:32:48 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 12:32:48 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 12:37:48 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 12:37:48 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 12:37:48 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 12:37:48 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 12:42:48 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 12:42:48 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 12:42:48 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 12:42:48 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 12:47:48 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 12:47:48 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 12:47:48 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 12:47:48 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 12:50:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  5 12:50:01 rdma06 dnf[462372]: Updating Subscription Management repositories.
Apr  5 12:50:01 rdma06 dnf[462372]: Unable to read consumer identity
Apr  5 12:50:01 rdma06 dnf[462372]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  5 12:50:02 rdma06 dnf[462372]: Copr repo for fmf owned by psss                 2.9 kB/s | 3.3 kB     00:01
Apr  5 12:50:02 rdma06 dnf[462372]: beaker-AppStream                                 53 kB/s | 3.2 kB     00:00
Apr  5 12:50:02 rdma06 dnf[462372]: beaker-AppStream-debuginfo                       45 kB/s | 1.5 kB     00:00
Apr  5 12:50:02 rdma06 dnf[462372]: beaker-BaseOS                                    80 kB/s | 2.7 kB     00:00
Apr  5 12:50:03 rdma06 dnf[462372]: beaker-BaseOS-debuginfo                          37 kB/s | 1.5 kB     00:00
Apr  5 12:50:03 rdma06 dnf[462372]: beaker-CRB                                       45 kB/s | 2.7 kB     00:00
Apr  5 12:50:03 rdma06 dnf[462372]: beaker-CRB-debuginfo                             59 kB/s | 1.5 kB     00:00
Apr  5 12:50:03 rdma06 dnf[462372]: beaker-HighAvailability                          52 kB/s | 2.8 kB     00:00
Apr  5 12:50:03 rdma06 dnf[462372]: beaker-HighAvailability-debuginfo                31 kB/s | 1.5 kB     00:00
Apr  5 12:50:03 rdma06 dnf[462372]: beaker-NFV                                       78 kB/s | 2.7 kB     00:00
Apr  5 12:50:03 rdma06 dnf[462372]: beaker-NFV-debuginfo                             48 kB/s | 1.5 kB     00:00
Apr  5 12:50:03 rdma06 dnf[462372]: beaker-RT                                        54 kB/s | 2.7 kB     00:00
Apr  5 12:50:03 rdma06 dnf[462372]: beaker-RT-debuginfo                              53 kB/s | 1.5 kB     00:00
Apr  5 12:50:03 rdma06 dnf[462372]: beaker-ResilientStorage                          65 kB/s | 2.8 kB     00:00
Apr  5 12:50:03 rdma06 dnf[462372]: beaker-ResilientStorage-debuginfo                50 kB/s | 1.5 kB     00:00
Apr  5 12:50:03 rdma06 dnf[462372]: beaker-SAP                                       69 kB/s | 2.7 kB     00:00
Apr  5 12:50:03 rdma06 dnf[462372]: beaker-SAP-debuginfo                             59 kB/s | 1.5 kB     00:00
Apr  5 12:50:03 rdma06 dnf[462372]: beaker-SAPHANA                                  123 kB/s | 2.7 kB     00:00
Apr  5 12:50:03 rdma06 dnf[462372]: beaker-SAPHANA-debuginfo                         32 kB/s | 1.5 kB     00:00
Apr  5 12:50:04 rdma06 dnf[462372]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr  5 12:50:05 rdma06 dnf[462372]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr  5 12:50:05 rdma06 dnf[462372]: Metadata cache created.
Apr  5 12:50:05 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  5 12:50:05 rdma06 systemd[1]: Finished dnf makecache.
Apr  5 12:50:05 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.382s CPU time.
Apr  5 12:52:48 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 12:52:48 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 12:52:48 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 12:52:48 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  5 12:57:48 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 12:57:48 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 12:57:48 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 12:57:48 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 13:02:48 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 13:02:48 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 13:02:48 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 13:02:48 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 13:07:48 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 13:07:48 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 13:07:48 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 13:07:48 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 13:12:49 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 13:12:49 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 13:12:49 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 13:12:49 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 13:17:49 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 13:17:49 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 13:17:49 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 13:17:49 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 13:22:49 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 13:22:49 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 13:22:49 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 13:22:49 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 13:27:49 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 13:27:49 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 13:27:49 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 13:27:49 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 13:32:49 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 13:32:49 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 13:32:49 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 13:32:49 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 13:37:49 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 13:37:49 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 13:37:49 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 13:37:49 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 13:42:49 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 13:42:49 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 13:42:49 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 13:42:49 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 13:47:49 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 13:47:49 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 13:47:49 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 13:47:49 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 13:52:49 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 13:52:49 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 13:52:49 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 13:52:49 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 13:53:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  5 13:53:01 rdma06 dnf[464932]: Updating Subscription Management repositories.
Apr  5 13:53:01 rdma06 dnf[464932]: Unable to read consumer identity
Apr  5 13:53:01 rdma06 dnf[464932]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  5 13:53:01 rdma06 dnf[464932]: Metadata cache refreshed recently.
Apr  5 13:53:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  5 13:53:01 rdma06 systemd[1]: Finished dnf makecache.
Apr  5 13:57:49 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 13:57:49 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 13:57:49 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 13:57:49 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 14:02:50 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 14:02:50 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 14:02:50 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 14:02:50 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 14:07:50 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 14:07:50 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 14:07:50 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 14:07:50 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 14:12:50 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 14:12:50 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 14:12:50 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 14:12:50 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 14:17:50 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 14:17:50 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 14:17:50 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 14:17:50 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  5 14:22:50 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 14:22:50 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 14:22:50 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 14:22:50 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 14:27:50 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 14:27:50 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 14:27:50 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 14:27:50 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 14:32:50 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 14:32:50 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 14:32:50 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 14:32:50 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.053 sec sweep
Apr  5 14:37:50 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 14:37:50 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 14:37:50 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 14:37:50 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 14:42:50 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 14:42:50 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 14:42:50 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 14:42:50 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 14:47:50 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 14:47:50 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 14:47:50 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 14:47:50 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 14:52:51 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 14:52:51 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 14:52:51 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 14:52:51 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 14:57:51 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 14:57:51 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 14:57:51 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 14:57:51 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 15:02:51 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 15:02:51 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 15:02:51 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 15:02:51 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 15:07:51 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 15:07:51 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 15:07:51 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 15:07:51 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 15:12:51 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 15:12:51 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 15:12:51 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 15:12:51 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 15:17:51 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 15:17:51 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 15:17:51 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 15:17:51 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 15:22:51 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 15:22:51 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 15:22:51 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 15:22:51 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 15:27:51 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 15:27:51 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 15:27:51 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 15:27:51 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 15:32:51 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 15:32:51 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 15:32:51 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 15:32:51 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 15:37:51 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 15:37:51 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 15:37:51 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 15:37:51 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 15:41:50 rdma06 systemd[1]: Starting dnf makecache...
Apr  5 15:41:51 rdma06 dnf[469339]: Updating Subscription Management repositories.
Apr  5 15:41:51 rdma06 dnf[469339]: Unable to read consumer identity
Apr  5 15:41:51 rdma06 dnf[469339]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  5 15:41:51 rdma06 dnf[469339]: Metadata cache refreshed recently.
Apr  5 15:41:51 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  5 15:41:51 rdma06 systemd[1]: Finished dnf makecache.
Apr  5 15:42:51 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 15:42:51 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 15:42:51 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 15:42:52 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 15:47:52 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 15:47:52 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 15:47:52 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 15:47:52 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 15:52:52 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 15:52:52 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 15:52:52 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 15:52:52 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 15:57:52 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 15:57:52 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 15:57:52 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 15:57:52 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 16:02:52 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 16:02:52 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 16:02:52 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 16:02:52 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  5 16:07:52 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 16:07:52 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 16:07:52 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 16:07:52 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 16:12:52 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 16:12:52 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 16:12:52 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 16:12:52 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 16:17:52 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 16:17:52 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 16:17:52 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 16:17:52 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 16:20:32 rdma06 NetworkManager[1025]: <info>  [1680726032.9331] dhcp4 (eno1): state changed new lease, address=10.73.131.45
Apr  5 16:22:52 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 16:22:52 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 16:22:52 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 16:22:52 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 16:27:52 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 16:27:52 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 16:27:52 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 16:27:52 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 16:32:52 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 16:32:52 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 16:32:52 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 16:32:53 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 16:37:53 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 16:37:53 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 16:37:53 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 16:37:53 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 16:42:53 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 16:42:53 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 16:42:53 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 16:42:53 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 16:47:53 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 16:47:53 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 16:47:53 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 16:47:53 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 16:50:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  5 16:50:01 rdma06 dnf[472096]: Updating Subscription Management repositories.
Apr  5 16:50:01 rdma06 dnf[472096]: Unable to read consumer identity
Apr  5 16:50:01 rdma06 dnf[472096]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  5 16:50:02 rdma06 dnf[472096]: Copr repo for fmf owned by psss                 2.5 kB/s | 3.3 kB     00:01
Apr  5 16:50:02 rdma06 dnf[472096]: beaker-AppStream                                 62 kB/s | 3.2 kB     00:00
Apr  5 16:50:03 rdma06 dnf[472096]: beaker-AppStream-debuginfo                       32 kB/s | 1.5 kB     00:00
Apr  5 16:50:03 rdma06 dnf[472096]: beaker-BaseOS                                   108 kB/s | 2.7 kB     00:00
Apr  5 16:50:03 rdma06 dnf[472096]: beaker-BaseOS-debuginfo                          34 kB/s | 1.5 kB     00:00
Apr  5 16:50:03 rdma06 dnf[472096]: beaker-CRB                                       72 kB/s | 2.7 kB     00:00
Apr  5 16:50:03 rdma06 dnf[472096]: beaker-CRB-debuginfo                             74 kB/s | 1.5 kB     00:00
Apr  5 16:50:03 rdma06 dnf[472096]: beaker-HighAvailability                          77 kB/s | 2.8 kB     00:00
Apr  5 16:50:03 rdma06 dnf[472096]: beaker-HighAvailability-debuginfo                34 kB/s | 1.5 kB     00:00
Apr  5 16:50:03 rdma06 dnf[472096]: beaker-NFV                                       72 kB/s | 2.7 kB     00:00
Apr  5 16:50:03 rdma06 dnf[472096]: beaker-NFV-debuginfo                             52 kB/s | 1.5 kB     00:00
Apr  5 16:50:03 rdma06 dnf[472096]: beaker-RT                                        41 kB/s | 2.7 kB     00:00
Apr  5 16:50:03 rdma06 dnf[472096]: beaker-RT-debuginfo                              69 kB/s | 1.5 kB     00:00
Apr  5 16:50:03 rdma06 dnf[472096]: beaker-ResilientStorage                          61 kB/s | 2.8 kB     00:00
Apr  5 16:50:03 rdma06 dnf[472096]: beaker-ResilientStorage-debuginfo                54 kB/s | 1.5 kB     00:00
Apr  5 16:50:03 rdma06 dnf[472096]: beaker-SAP                                      131 kB/s | 2.7 kB     00:00
Apr  5 16:50:03 rdma06 dnf[472096]: beaker-SAP-debuginfo                             59 kB/s | 1.5 kB     00:00
Apr  5 16:50:03 rdma06 dnf[472096]: beaker-SAPHANA                                   77 kB/s | 2.7 kB     00:00
Apr  5 16:50:03 rdma06 dnf[472096]: beaker-SAPHANA-debuginfo                         34 kB/s | 1.5 kB     00:00
Apr  5 16:50:04 rdma06 dnf[472096]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr  5 16:50:05 rdma06 dnf[472096]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr  5 16:50:05 rdma06 dnf[472096]: Metadata cache created.
Apr  5 16:50:05 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  5 16:50:05 rdma06 systemd[1]: Finished dnf makecache.
Apr  5 16:50:05 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.399s CPU time.
Apr  5 16:52:53 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 16:52:53 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 16:52:53 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 16:52:53 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  5 16:57:53 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 16:57:53 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 16:57:53 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 16:57:53 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  5 17:02:53 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 17:02:53 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 17:02:53 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 17:02:53 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 17:07:53 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 17:07:53 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 17:07:53 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 17:07:53 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 17:12:53 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 17:12:53 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 17:12:53 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 17:12:53 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 17:17:53 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 17:17:53 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 17:17:53 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 17:17:53 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 17:22:53 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 17:22:53 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 17:22:53 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 17:22:54 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 17:27:54 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 17:27:54 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 17:27:54 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 17:27:54 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 17:32:54 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 17:32:54 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 17:32:54 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 17:32:54 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 17:37:54 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 17:37:54 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 17:37:54 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 17:37:54 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 17:42:54 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 17:42:54 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 17:42:54 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 17:42:54 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 17:47:54 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 17:47:54 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 17:47:54 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 17:47:54 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 17:52:54 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 17:52:54 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 17:52:54 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 17:52:54 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 17:57:54 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 17:57:54 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 17:57:54 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 17:57:54 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 18:02:54 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 18:02:54 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 18:02:54 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 18:02:54 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 18:07:54 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 18:07:54 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 18:07:54 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 18:07:54 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  5 18:12:54 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 18:12:54 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 18:12:54 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 18:12:54 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 18:17:55 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 18:17:55 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 18:17:55 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 18:17:55 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 18:22:55 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 18:22:55 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 18:22:55 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 18:22:55 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 18:27:55 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 18:27:55 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 18:27:55 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 18:27:55 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 18:32:55 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 18:32:55 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 18:32:55 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 18:32:55 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 18:37:55 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 18:37:55 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 18:37:55 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 18:37:55 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 18:42:55 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 18:42:55 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 18:42:55 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 18:42:55 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 18:44:50 rdma06 systemd[1]: Starting dnf makecache...
Apr  5 18:44:51 rdma06 dnf[476764]: Updating Subscription Management repositories.
Apr  5 18:44:51 rdma06 dnf[476764]: Unable to read consumer identity
Apr  5 18:44:51 rdma06 dnf[476764]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  5 18:44:51 rdma06 dnf[476764]: Metadata cache refreshed recently.
Apr  5 18:44:51 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  5 18:44:51 rdma06 systemd[1]: Finished dnf makecache.
Apr  5 18:47:55 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 18:47:55 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 18:47:55 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 18:47:55 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  5 18:52:55 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 18:52:55 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 18:52:55 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 18:52:55 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 18:57:55 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 18:57:55 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 18:57:55 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 18:57:55 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 19:02:55 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 19:02:55 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 19:02:55 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 19:02:55 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 19:07:56 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 19:07:56 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 19:07:56 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 19:07:56 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 19:12:56 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 19:12:56 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 19:12:56 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 19:12:56 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 19:17:56 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 19:17:56 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 19:17:56 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 19:17:56 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 19:22:56 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 19:22:56 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 19:22:56 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 19:22:56 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 19:27:56 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 19:27:56 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 19:27:56 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 19:27:56 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 19:32:56 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 19:32:56 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 19:32:56 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 19:32:56 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  5 19:37:56 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 19:37:56 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 19:37:56 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 19:37:56 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 19:42:56 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 19:42:56 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 19:42:56 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 19:42:56 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 19:47:56 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 19:47:56 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 19:47:56 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 19:47:56 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 19:52:56 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 19:52:56 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 19:52:56 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 19:52:56 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 19:57:57 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 19:57:57 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 19:57:57 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 19:57:57 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  5 20:02:57 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 20:02:57 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 20:02:57 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 20:02:57 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 20:07:57 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 20:07:57 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 20:07:57 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 20:07:57 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 20:12:57 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 20:12:57 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 20:12:57 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 20:12:57 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 20:17:57 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 20:17:57 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 20:17:57 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 20:17:57 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 20:22:57 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 20:22:57 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 20:22:57 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 20:22:57 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 20:27:57 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 20:27:57 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 20:27:57 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 20:27:57 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.053 sec sweep
Apr  5 20:32:57 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 20:32:57 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 20:32:57 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 20:32:57 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.053 sec sweep
Apr  5 20:37:57 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 20:37:57 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 20:37:57 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 20:37:57 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.053 sec sweep
Apr  5 20:38:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  5 20:38:01 rdma06 dnf[481336]: Updating Subscription Management repositories.
Apr  5 20:38:01 rdma06 dnf[481336]: Unable to read consumer identity
Apr  5 20:38:01 rdma06 dnf[481336]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  5 20:38:02 rdma06 dnf[481336]: Copr repo for fmf owned by psss                 3.6 kB/s | 3.3 kB     00:00
Apr  5 20:38:02 rdma06 dnf[481336]: beaker-AppStream                                 40 kB/s | 3.2 kB     00:00
Apr  5 20:38:02 rdma06 dnf[481336]: beaker-AppStream-debuginfo                       27 kB/s | 1.5 kB     00:00
Apr  5 20:38:02 rdma06 dnf[481336]: beaker-BaseOS                                    63 kB/s | 2.7 kB     00:00
Apr  5 20:38:02 rdma06 dnf[481336]: beaker-BaseOS-debuginfo                          40 kB/s | 1.5 kB     00:00
Apr  5 20:38:02 rdma06 dnf[481336]: beaker-CRB                                       58 kB/s | 2.7 kB     00:00
Apr  5 20:38:02 rdma06 dnf[481336]: beaker-CRB-debuginfo                             62 kB/s | 1.5 kB     00:00
Apr  5 20:38:02 rdma06 dnf[481336]: beaker-HighAvailability                          56 kB/s | 2.8 kB     00:00
Apr  5 20:38:03 rdma06 dnf[481336]: beaker-HighAvailability-debuginfo                23 kB/s | 1.5 kB     00:00
Apr  5 20:38:03 rdma06 dnf[481336]: beaker-NFV                                       47 kB/s | 2.7 kB     00:00
Apr  5 20:38:03 rdma06 dnf[481336]: beaker-NFV-debuginfo                             22 kB/s | 1.5 kB     00:00
Apr  5 20:38:03 rdma06 dnf[481336]: beaker-RT                                        56 kB/s | 2.7 kB     00:00
Apr  5 20:38:03 rdma06 dnf[481336]: beaker-RT-debuginfo                              72 kB/s | 1.5 kB     00:00
Apr  5 20:38:03 rdma06 dnf[481336]: beaker-ResilientStorage                          72 kB/s | 2.8 kB     00:00
Apr  5 20:38:03 rdma06 dnf[481336]: beaker-ResilientStorage-debuginfo                48 kB/s | 1.5 kB     00:00
Apr  5 20:38:03 rdma06 dnf[481336]: beaker-SAP                                      193 kB/s | 2.7 kB     00:00
Apr  5 20:38:03 rdma06 dnf[481336]: beaker-SAP-debuginfo                             75 kB/s | 1.5 kB     00:00
Apr  5 20:38:03 rdma06 dnf[481336]: beaker-SAPHANA                                   55 kB/s | 2.7 kB     00:00
Apr  5 20:38:03 rdma06 dnf[481336]: beaker-SAPHANA-debuginfo                         37 kB/s | 1.5 kB     00:00
Apr  5 20:38:04 rdma06 dnf[481336]: beaker-harness                                  1.9 kB/s | 1.3 kB     00:00
Apr  5 20:38:04 rdma06 dnf[481336]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr  5 20:38:05 rdma06 dnf[481336]: Metadata cache created.
Apr  5 20:38:05 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  5 20:38:05 rdma06 systemd[1]: Finished dnf makecache.
Apr  5 20:38:05 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.383s CPU time.
Apr  5 20:42:57 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 20:42:57 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 20:42:57 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 20:42:57 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  5 20:47:57 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 20:47:57 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 20:47:57 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 20:47:58 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.053 sec sweep
Apr  5 20:52:58 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 20:52:58 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 20:52:58 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 20:52:58 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 20:57:58 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 20:57:58 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 20:57:58 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 20:57:58 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 21:02:58 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 21:02:58 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 21:02:58 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 21:02:58 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 21:07:58 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 21:07:58 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 21:07:58 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 21:07:58 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 21:12:58 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 21:12:58 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 21:12:58 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 21:12:58 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 21:17:58 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 21:17:58 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 21:17:58 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 21:17:58 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 21:22:58 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 21:22:58 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 21:22:58 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 21:22:58 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 21:27:58 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 21:27:58 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 21:27:58 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 21:27:58 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 21:32:58 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 21:32:58 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 21:32:58 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 21:32:58 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 21:37:58 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 21:37:58 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 21:37:58 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 21:37:58 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 21:42:50 rdma06 systemd[1]: Starting dnf makecache...
Apr  5 21:42:51 rdma06 dnf[483975]: Updating Subscription Management repositories.
Apr  5 21:42:51 rdma06 dnf[483975]: Unable to read consumer identity
Apr  5 21:42:51 rdma06 dnf[483975]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  5 21:42:51 rdma06 dnf[483975]: Metadata cache refreshed recently.
Apr  5 21:42:51 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  5 21:42:51 rdma06 systemd[1]: Finished dnf makecache.
Apr  5 21:42:59 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 21:42:59 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 21:42:59 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 21:42:59 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 21:47:59 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 21:47:59 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 21:47:59 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 21:47:59 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 21:52:59 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 21:52:59 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 21:52:59 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 21:52:59 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 21:57:59 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 21:57:59 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 21:57:59 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 21:57:59 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 22:02:59 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 22:02:59 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 22:02:59 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 22:02:59 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 22:07:59 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 22:07:59 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 22:07:59 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 22:07:59 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 22:12:59 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 22:12:59 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 22:12:59 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 22:12:59 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 22:17:59 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 22:17:59 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 22:17:59 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 22:17:59 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  5 22:20:32 rdma06 NetworkManager[1025]: <info>  [1680747632.9247] dhcp4 (eno1): state changed new lease, address=10.73.131.45
Apr  5 22:22:59 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 22:22:59 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 22:22:59 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 22:22:59 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 22:27:59 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 22:27:59 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 22:27:59 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 22:27:59 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 22:33:00 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 22:33:00 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 22:33:00 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 22:33:00 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 22:34:00 rdma06 systemd[5685]: Starting Cleanup of User's Temporary Files and Directories...
Apr  5 22:34:00 rdma06 systemd[5685]: Finished Cleanup of User's Temporary Files and Directories.
Apr  5 22:38:00 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 22:38:00 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 22:38:00 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 22:38:00 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 22:41:00 rdma06 systemd[1]: Starting Cleanup of Temporary Directories...
Apr  5 22:41:00 rdma06 systemd[1]: systemd-tmpfiles-clean.service: Deactivated successfully.
Apr  5 22:41:00 rdma06 systemd[1]: Finished Cleanup of Temporary Directories.
Apr  5 22:41:00 rdma06 systemd[1]: run-credentials-systemd\x2dtmpfiles\x2dclean.service.mount: Deactivated successfully.
Apr  5 22:43:00 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 22:43:00 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 22:43:00 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 22:43:00 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  5 22:48:00 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 22:48:00 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 22:48:00 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 22:48:00 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 22:53:00 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 22:53:00 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 22:53:00 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 22:53:00 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 22:58:00 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 22:58:00 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 22:58:00 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 22:58:00 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 23:03:00 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 23:03:00 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 23:03:00 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 23:03:00 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 23:08:00 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 23:08:00 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 23:08:00 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 23:08:00 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 23:13:00 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 23:13:00 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 23:13:00 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 23:13:00 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 23:18:00 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 23:18:00 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 23:18:00 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 23:18:00 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 23:23:00 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 23:23:00 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 23:23:00 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 23:23:01 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 23:28:01 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 23:28:01 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 23:28:01 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 23:28:01 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 23:32:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  5 23:32:01 rdma06 dnf[488391]: Updating Subscription Management repositories.
Apr  5 23:32:01 rdma06 dnf[488391]: Unable to read consumer identity
Apr  5 23:32:01 rdma06 dnf[488391]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  5 23:32:01 rdma06 dnf[488391]: Metadata cache refreshed recently.
Apr  5 23:32:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  5 23:32:01 rdma06 systemd[1]: Finished dnf makecache.
Apr  5 23:33:01 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 23:33:01 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 23:33:01 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 23:33:01 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  5 23:38:01 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 23:38:01 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 23:38:01 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 23:38:01 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 23:43:01 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 23:43:01 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 23:43:01 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 23:43:01 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 23:48:01 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 23:48:01 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 23:48:01 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 23:48:01 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 23:53:01 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 23:53:01 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 23:53:01 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 23:53:01 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  5 23:58:01 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  5 23:58:01 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  5 23:58:01 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  5 23:58:01 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 00:00:00 rdma06 systemd[1]: Starting Rotate log files...
Apr  6 00:00:00 rdma06 systemd[1]: logrotate.service: Deactivated successfully.
Apr  6 00:00:00 rdma06 systemd[1]: Finished Rotate log files.
Apr  6 00:03:01 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 00:03:01 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 00:03:01 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 00:03:01 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  6 00:08:01 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 00:08:01 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 00:08:01 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 00:08:01 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  6 00:13:01 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 00:13:01 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 00:13:01 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 00:13:02 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 00:18:02 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 00:18:02 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 00:18:02 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 00:18:02 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 00:23:02 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 00:23:02 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 00:23:02 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 00:23:02 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 00:28:02 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 00:28:02 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 00:28:02 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 00:28:02 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 00:33:02 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 00:33:02 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 00:33:02 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 00:33:02 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 00:38:02 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 00:38:02 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 00:38:02 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 00:38:02 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 00:43:02 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 00:43:02 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 00:43:02 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 00:43:02 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 00:48:02 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 00:48:02 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 00:48:02 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 00:48:02 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  6 00:52:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  6 00:52:01 rdma06 dnf[491637]: Updating Subscription Management repositories.
Apr  6 00:52:01 rdma06 dnf[491637]: Unable to read consumer identity
Apr  6 00:52:01 rdma06 dnf[491637]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  6 00:52:02 rdma06 dnf[491637]: Copr repo for fmf owned by psss                 3.5 kB/s | 3.3 kB     00:00
Apr  6 00:52:02 rdma06 dnf[491637]: beaker-AppStream                                 53 kB/s | 3.2 kB     00:00
Apr  6 00:52:02 rdma06 dnf[491637]: beaker-AppStream-debuginfo                       37 kB/s | 1.5 kB     00:00
Apr  6 00:52:02 rdma06 dnf[491637]: beaker-BaseOS                                    80 kB/s | 2.7 kB     00:00
Apr  6 00:52:02 rdma06 dnf[491637]: beaker-BaseOS-debuginfo                          41 kB/s | 1.5 kB     00:00
Apr  6 00:52:02 rdma06 dnf[491637]: beaker-CRB                                       63 kB/s | 2.7 kB     00:00
Apr  6 00:52:02 rdma06 dnf[491637]: beaker-CRB-debuginfo                             69 kB/s | 1.5 kB     00:00
Apr  6 00:52:02 rdma06 dnf[491637]: beaker-HighAvailability                          88 kB/s | 2.8 kB     00:00
Apr  6 00:52:03 rdma06 dnf[491637]: beaker-HighAvailability-debuginfo                45 kB/s | 1.5 kB     00:00
Apr  6 00:52:03 rdma06 dnf[491637]: beaker-NFV                                       76 kB/s | 2.7 kB     00:00
Apr  6 00:52:03 rdma06 dnf[491637]: beaker-NFV-debuginfo                             67 kB/s | 1.5 kB     00:00
Apr  6 00:52:03 rdma06 dnf[491637]: beaker-RT                                        60 kB/s | 2.7 kB     00:00
Apr  6 00:52:03 rdma06 dnf[491637]: beaker-RT-debuginfo                              91 kB/s | 1.5 kB     00:00
Apr  6 00:52:03 rdma06 dnf[491637]: beaker-ResilientStorage                          61 kB/s | 2.8 kB     00:00
Apr  6 00:52:03 rdma06 dnf[491637]: beaker-ResilientStorage-debuginfo                71 kB/s | 1.5 kB     00:00
Apr  6 00:52:03 rdma06 dnf[491637]: beaker-SAP                                      192 kB/s | 2.7 kB     00:00
Apr  6 00:52:03 rdma06 dnf[491637]: beaker-SAP-debuginfo                             41 kB/s | 1.5 kB     00:00
Apr  6 00:52:03 rdma06 dnf[491637]: beaker-SAPHANA                                   89 kB/s | 2.7 kB     00:00
Apr  6 00:52:03 rdma06 dnf[491637]: beaker-SAPHANA-debuginfo                         46 kB/s | 1.5 kB     00:00
Apr  6 00:52:04 rdma06 dnf[491637]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr  6 00:52:04 rdma06 dnf[491637]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr  6 00:52:05 rdma06 dnf[491637]: Metadata cache created.
Apr  6 00:52:05 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  6 00:52:05 rdma06 systemd[1]: Finished dnf makecache.
Apr  6 00:52:05 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.387s CPU time.
Apr  6 00:53:02 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 00:53:02 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 00:53:02 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 00:53:02 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  6 00:58:02 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 00:58:02 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 00:58:02 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 00:58:02 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 01:03:02 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 01:03:02 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 01:03:02 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 01:03:03 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  6 01:08:03 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 01:08:03 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 01:08:03 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 01:08:03 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 01:13:03 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 01:13:03 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 01:13:03 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 01:13:03 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 01:18:03 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 01:18:03 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 01:18:03 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 01:18:03 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 01:23:03 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 01:23:03 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 01:23:03 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 01:23:03 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 01:28:03 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 01:28:03 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 01:28:03 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 01:28:03 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 01:32:38 rdma06 chronyd[1034]: Selected source 10.18.100.10
Apr  6 01:33:03 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 01:33:03 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 01:33:03 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 01:33:03 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 01:38:03 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 01:38:03 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 01:38:03 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 01:38:03 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.053 sec sweep
Apr  6 01:43:03 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 01:43:03 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 01:43:03 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 01:43:03 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 01:48:03 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 01:48:03 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 01:48:03 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 01:48:03 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.053 sec sweep
Apr  6 01:53:03 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 01:53:03 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 01:53:03 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 01:53:04 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 01:58:04 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 01:58:04 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 01:58:04 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 01:58:04 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.053 sec sweep
Apr  6 02:03:04 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 02:03:04 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 02:03:04 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 02:03:04 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 02:08:04 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 02:08:04 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 02:08:04 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 02:08:04 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 02:10:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  6 02:10:01 rdma06 dnf[494825]: Updating Subscription Management repositories.
Apr  6 02:10:01 rdma06 dnf[494825]: Unable to read consumer identity
Apr  6 02:10:01 rdma06 dnf[494825]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  6 02:10:01 rdma06 dnf[494825]: Metadata cache refreshed recently.
Apr  6 02:10:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  6 02:10:01 rdma06 systemd[1]: Finished dnf makecache.
Apr  6 02:13:04 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 02:13:04 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 02:13:04 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 02:13:04 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  6 02:18:04 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 02:18:04 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 02:18:04 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 02:18:04 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  6 02:19:41 rdma06 chronyd[1034]: Selected source 10.11.160.238
Apr  6 02:23:04 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 02:23:04 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 02:23:04 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 02:23:04 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 02:28:04 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 02:28:04 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 02:28:04 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 02:28:04 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 02:33:04 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 02:33:04 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 02:33:04 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 02:33:04 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 02:38:04 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 02:38:04 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 02:38:04 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 02:38:04 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 02:43:04 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 02:43:04 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 02:43:04 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 02:43:04 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 02:48:05 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 02:48:05 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 02:48:05 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 02:48:05 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 02:53:05 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 02:53:05 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 02:53:05 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 02:53:05 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 02:58:05 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 02:58:05 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 02:58:05 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 02:58:05 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 03:03:05 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 03:03:05 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 03:03:05 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 03:03:05 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 03:08:05 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 03:08:05 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 03:08:05 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 03:08:05 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 03:13:05 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 03:13:05 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 03:13:05 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 03:13:05 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 03:18:05 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 03:18:05 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 03:18:05 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 03:18:05 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 03:23:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  6 03:23:01 rdma06 dnf[497781]: Updating Subscription Management repositories.
Apr  6 03:23:01 rdma06 dnf[497781]: Unable to read consumer identity
Apr  6 03:23:01 rdma06 dnf[497781]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  6 03:23:01 rdma06 dnf[497781]: Metadata cache refreshed recently.
Apr  6 03:23:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  6 03:23:01 rdma06 systemd[1]: Finished dnf makecache.
Apr  6 03:23:05 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 03:23:05 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 03:23:05 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 03:23:05 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 03:28:05 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 03:28:05 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 03:28:05 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 03:28:05 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  6 03:33:05 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 03:33:05 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 03:33:05 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 03:33:05 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 03:38:05 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 03:38:05 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 03:38:05 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 03:38:06 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 03:43:06 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 03:43:06 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 03:43:06 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 03:43:06 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  6 03:48:06 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 03:48:06 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 03:48:06 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 03:48:06 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 03:53:06 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 03:53:06 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 03:53:06 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 03:53:06 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 03:58:06 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 03:58:06 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 03:58:06 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 03:58:06 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 04:03:06 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 04:03:06 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 04:03:06 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 04:03:06 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 04:08:06 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 04:08:06 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 04:08:06 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 04:08:06 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 04:11:49 rdma06 chronyd[1034]: Selected source 10.18.100.10
Apr  6 04:13:06 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 04:13:06 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 04:13:06 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 04:13:06 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 04:18:06 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 04:18:06 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 04:18:06 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 04:18:06 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 04:20:32 rdma06 NetworkManager[1025]: <info>  [1680769232.9135] dhcp4 (eno1): state changed new lease, address=10.73.131.45
Apr  6 04:23:06 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 04:23:06 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 04:23:06 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 04:23:06 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 04:28:06 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 04:28:06 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 04:28:06 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 04:28:07 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 04:33:07 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 04:33:07 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 04:33:07 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 04:33:07 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.056 sec sweep
Apr  6 04:36:46 rdma06 sshd[500767]: main: sshd: ssh-rsa algorithm is disabled
Apr  6 04:36:57 rdma06 systemd-logind[1064]: New session 9 of user root.
Apr  6 04:36:57 rdma06 systemd[1]: Started Session 9 of User root.
Apr  6 04:36:57 rdma06 systemd[1]: Starting Hostname Service...
Apr  6 04:36:57 rdma06 systemd[1]: Started Hostname Service.
Apr  6 04:37:00 rdma06 journal[2657]: Adjustment to local watchdog ignored since 'no_localwatchdog' metadata is set
Apr  6 04:37:27 rdma06 systemd[1]: systemd-hostnamed.service: Deactivated successfully.
Apr  6 04:38:07 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 04:38:07 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 04:38:07 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 04:38:07 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.056 sec sweep
Apr  6 04:43:07 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 04:43:07 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 04:43:07 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 04:43:07 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.056 sec sweep
Apr  6 04:44:01 rdma06 systemd[5685]: Starting D-Bus User Message Bus...
Apr  6 04:44:01 rdma06 dbus-broker-launch[501175]: Policy to allow eavesdropping in /usr/share/dbus-1/session.conf +31: Eavesdropping is deprecated and ignored
Apr  6 04:44:01 rdma06 dbus-broker-launch[501175]: Policy to allow eavesdropping in /usr/share/dbus-1/session.conf +33: Eavesdropping is deprecated and ignored
Apr  6 04:44:01 rdma06 systemd[5685]: Started D-Bus User Message Bus.
Apr  6 04:44:01 rdma06 journal[501175]: Ready
Apr  6 04:48:07 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 04:48:07 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 04:48:07 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 04:48:07 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.056 sec sweep
Apr  6 04:50:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  6 04:50:01 rdma06 dnf[501453]: Updating Subscription Management repositories.
Apr  6 04:50:01 rdma06 dnf[501453]: Unable to read consumer identity
Apr  6 04:50:01 rdma06 dnf[501453]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  6 04:50:03 rdma06 dnf[501453]: Copr repo for fmf owned by psss                 2.4 kB/s | 3.3 kB     00:01
Apr  6 04:50:03 rdma06 dnf[501453]: beaker-AppStream                                 42 kB/s | 3.2 kB     00:00
Apr  6 04:50:03 rdma06 dnf[501453]: beaker-AppStream-debuginfo                       42 kB/s | 1.5 kB     00:00
Apr  6 04:50:03 rdma06 dnf[501453]: beaker-BaseOS                                    53 kB/s | 2.7 kB     00:00
Apr  6 04:50:03 rdma06 dnf[501453]: beaker-BaseOS-debuginfo                          47 kB/s | 1.5 kB     00:00
Apr  6 04:50:03 rdma06 dnf[501453]: beaker-CRB                                       60 kB/s | 2.7 kB     00:00
Apr  6 04:50:03 rdma06 dnf[501453]: beaker-CRB-debuginfo                             57 kB/s | 1.5 kB     00:00
Apr  6 04:50:03 rdma06 dnf[501453]: beaker-HighAvailability                          56 kB/s | 2.8 kB     00:00
Apr  6 04:50:03 rdma06 dnf[501453]: beaker-HighAvailability-debuginfo                41 kB/s | 1.5 kB     00:00
Apr  6 04:50:03 rdma06 dnf[501453]: beaker-NFV                                       74 kB/s | 2.7 kB     00:00
Apr  6 04:50:03 rdma06 dnf[501453]: beaker-NFV-debuginfo                             54 kB/s | 1.5 kB     00:00
Apr  6 04:50:03 rdma06 dnf[501453]: beaker-RT                                        66 kB/s | 2.7 kB     00:00
Apr  6 04:50:03 rdma06 dnf[501453]: beaker-RT-debuginfo                              82 kB/s | 1.5 kB     00:00
Apr  6 04:50:03 rdma06 dnf[501453]: beaker-ResilientStorage                          71 kB/s | 2.8 kB     00:00
Apr  6 04:50:03 rdma06 dnf[501453]: beaker-ResilientStorage-debuginfo                69 kB/s | 1.5 kB     00:00
Apr  6 04:50:03 rdma06 dnf[501453]: beaker-SAP                                       71 kB/s | 2.7 kB     00:00
Apr  6 04:50:03 rdma06 dnf[501453]: beaker-SAP-debuginfo                             59 kB/s | 1.5 kB     00:00
Apr  6 04:50:03 rdma06 dnf[501453]: beaker-SAPHANA                                   93 kB/s | 2.7 kB     00:00
Apr  6 04:50:04 rdma06 dnf[501453]: beaker-SAPHANA-debuginfo                         46 kB/s | 1.5 kB     00:00
Apr  6 04:50:04 rdma06 dnf[501453]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr  6 04:50:05 rdma06 dnf[501453]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr  6 04:50:05 rdma06 dnf[501453]: Metadata cache created.
Apr  6 04:50:05 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  6 04:50:05 rdma06 systemd[1]: Finished dnf makecache.
Apr  6 04:50:05 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.397s CPU time.
Apr  6 04:53:07 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 04:53:07 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 04:53:07 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 04:53:07 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.056 sec sweep
Apr  6 04:58:07 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 04:58:07 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 04:58:07 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 04:58:07 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  6 05:03:07 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 05:03:07 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 05:03:07 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 05:03:07 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 05:06:29 rdma06 systemd[1]: Starting SSSD Kerberos Cache Manager...
Apr  6 05:06:30 rdma06 systemd[1]: Started SSSD Kerberos Cache Manager.
Apr  6 05:06:30 rdma06 sssd_kcm[502158]: Starting up
Apr  6 05:08:07 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 05:08:07 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 05:08:07 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 05:08:07 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  6 05:13:07 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 05:13:07 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 05:13:07 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 05:13:07 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  6 05:14:00 rdma06 sssd_kcm[502158]: Shutting down (status = 0)
Apr  6 05:14:00 rdma06 systemd[1]: sssd-kcm.service: Deactivated successfully.
Apr  6 05:18:07 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 05:18:07 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 05:18:07 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 05:18:08 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  6 05:23:08 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 05:23:08 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 05:23:08 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 05:23:08 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 05:28:08 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 05:28:08 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 05:28:08 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 05:28:08 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 05:33:08 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 05:33:08 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 05:33:08 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 05:33:08 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 05:38:08 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 05:38:08 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 05:38:08 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 05:38:08 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 05:43:08 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 05:43:08 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 05:43:08 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 05:43:08 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 05:48:08 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 05:48:08 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 05:48:08 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 05:48:08 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 05:53:08 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 05:53:08 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 05:53:08 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 05:53:08 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 05:56:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  6 05:56:01 rdma06 dnf[504172]: Updating Subscription Management repositories.
Apr  6 05:56:01 rdma06 dnf[504172]: Unable to read consumer identity
Apr  6 05:56:01 rdma06 dnf[504172]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  6 05:56:01 rdma06 dnf[504172]: Metadata cache refreshed recently.
Apr  6 05:56:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  6 05:56:01 rdma06 systemd[1]: Finished dnf makecache.
Apr  6 05:58:08 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 05:58:08 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 05:58:08 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 05:58:08 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  6 05:59:58 rdma06 chronyd[1034]: Selected source 10.11.160.238
Apr  6 06:03:08 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 06:03:08 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 06:03:08 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 06:03:08 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 06:08:08 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 06:08:08 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 06:08:08 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 06:08:08 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 06:13:09 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 06:13:09 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 06:13:09 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 06:13:09 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 06:18:09 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 06:18:09 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 06:18:09 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 06:18:09 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  6 06:23:09 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 06:23:09 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 06:23:09 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 06:23:09 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 06:28:09 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 06:28:09 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 06:28:09 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 06:28:09 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 06:33:09 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 06:33:09 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 06:33:09 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 06:33:09 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 06:38:09 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 06:38:09 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 06:38:09 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 06:38:09 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 06:43:09 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 06:43:09 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 06:43:09 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 06:43:09 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  6 06:48:09 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 06:48:09 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 06:48:09 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 06:48:09 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 06:53:09 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 06:53:09 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 06:53:09 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 06:53:09 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 06:58:09 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 06:58:09 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 06:58:09 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 06:58:09 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 07:03:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 07:03:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 07:03:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 07:03:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 07:08:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 07:08:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 07:08:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 07:08:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 07:13:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 07:13:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 07:13:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 07:13:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 07:18:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 07:18:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 07:18:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 07:18:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 07:20:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  6 07:20:01 rdma06 dnf[507568]: Updating Subscription Management repositories.
Apr  6 07:20:01 rdma06 dnf[507568]: Unable to read consumer identity
Apr  6 07:20:01 rdma06 dnf[507568]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  6 07:20:01 rdma06 dnf[507568]: Metadata cache refreshed recently.
Apr  6 07:20:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  6 07:20:01 rdma06 systemd[1]: Finished dnf makecache.
Apr  6 07:23:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 07:23:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 07:23:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 07:23:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 07:28:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 07:28:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 07:28:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 07:28:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.053 sec sweep
Apr  6 07:33:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 07:33:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 07:33:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 07:33:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 07:38:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 07:38:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 07:38:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 07:38:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.053 sec sweep
Apr  6 07:43:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 07:43:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 07:43:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 07:43:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 07:48:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 07:48:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 07:48:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 07:48:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 07:53:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 07:53:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 07:53:10 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 07:53:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.053 sec sweep
Apr  6 07:58:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 07:58:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 07:58:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 07:58:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.053 sec sweep
Apr  6 08:03:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 08:03:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 08:03:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 08:03:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.053 sec sweep
Apr  6 08:08:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 08:08:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 08:08:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 08:08:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 08:13:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 08:13:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 08:13:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 08:13:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.053 sec sweep
Apr  6 08:18:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 08:18:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 08:18:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 08:18:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 08:23:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 08:23:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 08:23:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 08:23:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 08:28:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 08:28:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 08:28:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 08:28:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.053 sec sweep
Apr  6 08:33:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 08:33:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 08:33:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 08:33:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 08:38:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 08:38:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 08:38:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 08:38:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 08:40:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  6 08:40:01 rdma06 dnf[510791]: Updating Subscription Management repositories.
Apr  6 08:40:01 rdma06 dnf[510791]: Unable to read consumer identity
Apr  6 08:40:01 rdma06 dnf[510791]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  6 08:40:04 rdma06 dnf[510791]: Copr repo for fmf owned by psss                 1.3 kB/s | 3.3 kB     00:02
Apr  6 08:40:04 rdma06 dnf[510791]: beaker-AppStream                                 52 kB/s | 3.2 kB     00:00
Apr  6 08:40:04 rdma06 dnf[510791]: beaker-AppStream-debuginfo                       37 kB/s | 1.5 kB     00:00
Apr  6 08:40:04 rdma06 dnf[510791]: beaker-BaseOS                                    77 kB/s | 2.7 kB     00:00
Apr  6 08:40:04 rdma06 dnf[510791]: beaker-BaseOS-debuginfo                          41 kB/s | 1.5 kB     00:00
Apr  6 08:40:04 rdma06 dnf[510791]: beaker-CRB                                       64 kB/s | 2.7 kB     00:00
Apr  6 08:40:04 rdma06 dnf[510791]: beaker-CRB-debuginfo                             64 kB/s | 1.5 kB     00:00
Apr  6 08:40:04 rdma06 dnf[510791]: beaker-HighAvailability                          78 kB/s | 2.8 kB     00:00
Apr  6 08:40:04 rdma06 dnf[510791]: beaker-HighAvailability-debuginfo                41 kB/s | 1.5 kB     00:00
Apr  6 08:40:04 rdma06 dnf[510791]: beaker-NFV                                       70 kB/s | 2.7 kB     00:00
Apr  6 08:40:04 rdma06 dnf[510791]: beaker-NFV-debuginfo                             60 kB/s | 1.5 kB     00:00
Apr  6 08:40:04 rdma06 dnf[510791]: beaker-RT                                        62 kB/s | 2.7 kB     00:00
Apr  6 08:40:04 rdma06 dnf[510791]: beaker-RT-debuginfo                              74 kB/s | 1.5 kB     00:00
Apr  6 08:40:04 rdma06 dnf[510791]: beaker-ResilientStorage                          75 kB/s | 2.8 kB     00:00
Apr  6 08:40:04 rdma06 dnf[510791]: beaker-ResilientStorage-debuginfo                64 kB/s | 1.5 kB     00:00
Apr  6 08:40:04 rdma06 dnf[510791]: beaker-SAP                                       81 kB/s | 2.7 kB     00:00
Apr  6 08:40:04 rdma06 dnf[510791]: beaker-SAP-debuginfo                             76 kB/s | 1.5 kB     00:00
Apr  6 08:40:04 rdma06 dnf[510791]: beaker-SAPHANA                                   85 kB/s | 2.7 kB     00:00
Apr  6 08:40:04 rdma06 dnf[510791]: beaker-SAPHANA-debuginfo                         39 kB/s | 1.5 kB     00:00
Apr  6 08:40:05 rdma06 dnf[510791]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr  6 08:40:06 rdma06 dnf[510791]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr  6 08:40:06 rdma06 dnf[510791]: Metadata cache created.
Apr  6 08:40:06 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  6 08:40:06 rdma06 systemd[1]: Finished dnf makecache.
Apr  6 08:40:06 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.356s CPU time.
Apr  6 08:43:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 08:43:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 08:43:11 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 08:43:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  6 08:48:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 08:48:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 08:48:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 08:48:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 08:53:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 08:53:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 08:53:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 08:53:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 08:58:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 08:58:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 08:58:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 08:58:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 09:03:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 09:03:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 09:03:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 09:03:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 09:08:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 09:08:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 09:08:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 09:08:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 09:13:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 09:13:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 09:13:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 09:13:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  6 09:18:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 09:18:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 09:18:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 09:18:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 09:23:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 09:23:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 09:23:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 09:23:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 09:26:57 rdma06 chronyd[1034]: Selected source 10.18.100.10
Apr  6 09:28:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 09:28:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 09:28:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 09:28:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 09:33:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 09:33:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 09:33:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 09:33:12 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 09:38:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 09:38:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 09:38:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 09:38:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 09:43:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 09:43:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 09:43:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 09:43:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 09:48:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 09:48:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 09:48:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 09:48:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 09:53:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 09:53:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 09:53:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 09:53:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 09:58:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 09:58:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 09:58:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 09:58:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 10:03:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 10:03:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 10:03:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 10:03:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 10:08:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 10:08:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 10:08:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 10:08:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 10:13:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 10:13:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 10:13:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 10:13:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 10:18:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 10:18:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 10:18:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 10:18:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  6 10:20:32 rdma06 NetworkManager[1025]: <info>  [1680790832.9715] dhcp4 (eno1): state changed new lease, address=10.73.131.45
Apr  6 10:23:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 10:23:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 10:23:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 10:23:13 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 10:23:14 rdma06 systemd[1]: Starting dnf makecache...
Apr  6 10:23:14 rdma06 dnf[514985]: Updating Subscription Management repositories.
Apr  6 10:23:14 rdma06 dnf[514985]: Unable to read consumer identity
Apr  6 10:23:14 rdma06 dnf[514985]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  6 10:23:14 rdma06 dnf[514985]: Metadata cache refreshed recently.
Apr  6 10:23:14 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  6 10:23:14 rdma06 systemd[1]: Finished dnf makecache.
Apr  6 10:28:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 10:28:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 10:28:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 10:28:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 10:33:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 10:33:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 10:33:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 10:33:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 10:38:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 10:38:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 10:38:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 10:38:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 10:43:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 10:43:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 10:43:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 10:43:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 10:48:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 10:48:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 10:48:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 10:48:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 10:53:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 10:53:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 10:53:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 10:53:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 10:58:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 10:58:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 10:58:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 10:58:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 11:03:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 11:03:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 11:03:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 11:03:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 11:08:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 11:08:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 11:08:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 11:08:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 11:13:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 11:13:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 11:13:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 11:13:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 11:18:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 11:18:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 11:18:14 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 11:18:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 11:23:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 11:23:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 11:23:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 11:23:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 11:28:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 11:28:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 11:28:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 11:28:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 11:29:50 rdma06 systemd[1]: Starting dnf makecache...
Apr  6 11:29:51 rdma06 dnf[517670]: Updating Subscription Management repositories.
Apr  6 11:29:51 rdma06 dnf[517670]: Unable to read consumer identity
Apr  6 11:29:51 rdma06 dnf[517670]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  6 11:29:51 rdma06 dnf[517670]: Metadata cache refreshed recently.
Apr  6 11:29:51 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  6 11:29:51 rdma06 systemd[1]: Finished dnf makecache.
Apr  6 11:33:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 11:33:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 11:33:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 11:33:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 11:38:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 11:38:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 11:38:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 11:38:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 11:43:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 11:43:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 11:43:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 11:43:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 11:48:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 11:48:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 11:48:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 11:48:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 11:53:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 11:53:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 11:53:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 11:53:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 11:58:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 11:58:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 11:58:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 11:58:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 12:03:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 12:03:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 12:03:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 12:03:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 12:08:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 12:08:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 12:08:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 12:08:15 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 12:13:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 12:13:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 12:13:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 12:13:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 12:18:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 12:18:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 12:18:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 12:18:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 12:23:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 12:23:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 12:23:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 12:23:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 12:28:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 12:28:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 12:28:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 12:28:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 12:33:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 12:33:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 12:33:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 12:33:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 12:38:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 12:38:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 12:38:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 12:38:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 12:43:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 12:43:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 12:43:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 12:43:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 12:48:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 12:48:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 12:48:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 12:48:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 12:53:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 12:53:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 12:53:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 12:53:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 12:58:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 12:58:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 12:58:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 12:58:16 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 13:03:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 13:03:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 13:03:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 13:03:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 13:08:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 13:08:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 13:08:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 13:08:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 13:13:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 13:13:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 13:13:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 13:13:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 13:18:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 13:18:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 13:18:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 13:18:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 13:19:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  6 13:19:01 rdma06 dnf[522076]: Updating Subscription Management repositories.
Apr  6 13:19:01 rdma06 dnf[522076]: Unable to read consumer identity
Apr  6 13:19:01 rdma06 dnf[522076]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  6 13:19:02 rdma06 dnf[522076]: Copr repo for fmf owned by psss                 2.6 kB/s | 3.3 kB     00:01
Apr  6 13:19:02 rdma06 dnf[522076]: beaker-AppStream                                 84 kB/s | 3.2 kB     00:00
Apr  6 13:19:03 rdma06 dnf[522076]: beaker-AppStream-debuginfo                       47 kB/s | 1.5 kB     00:00
Apr  6 13:19:03 rdma06 dnf[522076]: beaker-BaseOS                                   111 kB/s | 2.7 kB     00:00
Apr  6 13:19:03 rdma06 dnf[522076]: beaker-BaseOS-debuginfo                          38 kB/s | 1.5 kB     00:00
Apr  6 13:19:03 rdma06 dnf[522076]: beaker-CRB                                       67 kB/s | 2.7 kB     00:00
Apr  6 13:19:03 rdma06 dnf[522076]: beaker-CRB-debuginfo                             58 kB/s | 1.5 kB     00:00
Apr  6 13:19:03 rdma06 dnf[522076]: beaker-HighAvailability                          63 kB/s | 2.8 kB     00:00
Apr  6 13:19:03 rdma06 dnf[522076]: beaker-HighAvailability-debuginfo                43 kB/s | 1.5 kB     00:00
Apr  6 13:19:03 rdma06 dnf[522076]: beaker-NFV                                       80 kB/s | 2.7 kB     00:00
Apr  6 13:19:03 rdma06 dnf[522076]: beaker-NFV-debuginfo                             57 kB/s | 1.5 kB     00:00
Apr  6 13:19:03 rdma06 dnf[522076]: beaker-RT                                        54 kB/s | 2.7 kB     00:00
Apr  6 13:19:03 rdma06 dnf[522076]: beaker-RT-debuginfo                              40 kB/s | 1.5 kB     00:00
Apr  6 13:19:03 rdma06 dnf[522076]: beaker-ResilientStorage                          70 kB/s | 2.8 kB     00:00
Apr  6 13:19:03 rdma06 dnf[522076]: beaker-ResilientStorage-debuginfo                55 kB/s | 1.5 kB     00:00
Apr  6 13:19:03 rdma06 dnf[522076]: beaker-SAP                                       89 kB/s | 2.7 kB     00:00
Apr  6 13:19:03 rdma06 dnf[522076]: beaker-SAP-debuginfo                             76 kB/s | 1.5 kB     00:00
Apr  6 13:19:03 rdma06 dnf[522076]: beaker-SAPHANA                                   77 kB/s | 2.7 kB     00:00
Apr  6 13:19:03 rdma06 dnf[522076]: beaker-SAPHANA-debuginfo                         37 kB/s | 1.5 kB     00:00
Apr  6 13:19:04 rdma06 dnf[522076]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr  6 13:19:05 rdma06 dnf[522076]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr  6 13:19:05 rdma06 dnf[522076]: Metadata cache created.
Apr  6 13:19:05 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  6 13:19:05 rdma06 systemd[1]: Finished dnf makecache.
Apr  6 13:19:05 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.415s CPU time.
Apr  6 13:23:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 13:23:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 13:23:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 13:23:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  6 13:28:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 13:28:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 13:28:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 13:28:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 13:33:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 13:33:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 13:33:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 13:33:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.056 sec sweep
Apr  6 13:38:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 13:38:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 13:38:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 13:38:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 13:43:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 13:43:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 13:43:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 13:43:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 13:48:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 13:48:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 13:48:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 13:48:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 13:53:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 13:53:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 13:53:17 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 13:53:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 13:58:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 13:58:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 13:58:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 13:58:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 14:03:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 14:03:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 14:03:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 14:03:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  6 14:08:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 14:08:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 14:08:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 14:08:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 14:13:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 14:13:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 14:13:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 14:13:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 14:18:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 14:18:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 14:18:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 14:18:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.056 sec sweep
Apr  6 14:23:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 14:23:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 14:23:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 14:23:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 14:28:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 14:28:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 14:28:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 14:28:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 14:33:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 14:33:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 14:33:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 14:33:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 14:38:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 14:38:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 14:38:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 14:38:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 14:43:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 14:43:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 14:43:18 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 14:43:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 14:48:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 14:48:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 14:48:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 14:48:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 14:52:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  6 14:52:01 rdma06 dnf[525845]: Updating Subscription Management repositories.
Apr  6 14:52:01 rdma06 dnf[525845]: Unable to read consumer identity
Apr  6 14:52:01 rdma06 dnf[525845]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  6 14:52:01 rdma06 dnf[525845]: Metadata cache refreshed recently.
Apr  6 14:52:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  6 14:52:01 rdma06 systemd[1]: Finished dnf makecache.
Apr  6 14:53:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 14:53:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 14:53:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 14:53:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 14:58:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 14:58:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 14:58:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 14:58:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  6 15:03:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 15:03:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 15:03:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 15:03:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 15:08:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 15:08:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 15:08:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 15:08:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 15:13:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 15:13:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 15:13:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 15:13:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 15:18:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 15:18:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 15:18:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 15:18:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 15:23:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 15:23:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 15:23:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 15:23:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 15:28:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 15:28:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 15:28:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 15:28:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 15:33:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 15:33:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 15:33:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 15:33:19 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 15:38:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 15:38:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 15:38:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 15:38:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  6 15:43:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 15:43:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 15:43:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 15:43:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 15:48:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 15:48:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 15:48:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 15:48:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 15:53:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 15:53:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 15:53:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 15:53:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 15:58:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 15:58:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 15:58:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 15:58:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 16:03:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 16:03:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 16:03:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 16:03:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 16:08:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 16:08:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 16:08:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 16:08:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.053 sec sweep
Apr  6 16:13:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 16:13:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 16:13:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 16:13:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 16:17:50 rdma06 systemd[1]: Starting dnf makecache...
Apr  6 16:17:51 rdma06 dnf[529318]: Updating Subscription Management repositories.
Apr  6 16:17:51 rdma06 dnf[529318]: Unable to read consumer identity
Apr  6 16:17:51 rdma06 dnf[529318]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  6 16:17:51 rdma06 dnf[529318]: Metadata cache refreshed recently.
Apr  6 16:17:51 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  6 16:17:51 rdma06 systemd[1]: Finished dnf makecache.
Apr  6 16:18:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 16:18:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 16:18:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 16:18:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 16:20:32 rdma06 NetworkManager[1025]: <info>  [1680812432.8690] dhcp4 (eno1): state changed new lease, address=10.73.131.45
Apr  6 16:23:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 16:23:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 16:23:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 16:23:20 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 16:28:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 16:28:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 16:28:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 16:28:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 16:33:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 16:33:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 16:33:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 16:33:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.053 sec sweep
Apr  6 16:38:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 16:38:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 16:38:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 16:38:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 16:43:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 16:43:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 16:43:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 16:43:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 16:48:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 16:48:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 16:48:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 16:48:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 16:53:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 16:53:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 16:53:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 16:53:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.053 sec sweep
Apr  6 16:58:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 16:58:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 16:58:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 16:58:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 17:03:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 17:03:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 17:03:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 17:03:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 17:08:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 17:08:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 17:08:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 17:08:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 17:13:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 17:13:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 17:13:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 17:13:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 17:18:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 17:18:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 17:18:21 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 17:18:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  6 17:23:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 17:23:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 17:23:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 17:23:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  6 17:28:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 17:28:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 17:28:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 17:28:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 17:33:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 17:33:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 17:33:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 17:33:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 17:38:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 17:38:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 17:38:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 17:38:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 17:43:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 17:43:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 17:43:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 17:43:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 17:48:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 17:48:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 17:48:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 17:48:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 17:53:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 17:53:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 17:53:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 17:53:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 17:56:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  6 17:56:01 rdma06 dnf[533272]: Updating Subscription Management repositories.
Apr  6 17:56:01 rdma06 dnf[533272]: Unable to read consumer identity
Apr  6 17:56:01 rdma06 dnf[533272]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  6 17:56:02 rdma06 dnf[533272]: Copr repo for fmf owned by psss                 2.4 kB/s | 3.3 kB     00:01
Apr  6 17:56:03 rdma06 dnf[533272]: beaker-AppStream                                 25 kB/s | 3.2 kB     00:00
Apr  6 17:56:03 rdma06 dnf[533272]: beaker-AppStream-debuginfo                       34 kB/s | 1.5 kB     00:00
Apr  6 17:56:03 rdma06 dnf[533272]: beaker-BaseOS                                    26 kB/s | 2.7 kB     00:00
Apr  6 17:56:03 rdma06 dnf[533272]: beaker-BaseOS-debuginfo                          24 kB/s | 1.5 kB     00:00
Apr  6 17:56:03 rdma06 dnf[533272]: beaker-CRB                                       55 kB/s | 2.7 kB     00:00
Apr  6 17:56:03 rdma06 dnf[533272]: beaker-CRB-debuginfo                             70 kB/s | 1.5 kB     00:00
Apr  6 17:56:03 rdma06 dnf[533272]: beaker-HighAvailability                          69 kB/s | 2.8 kB     00:00
Apr  6 17:56:03 rdma06 dnf[533272]: beaker-HighAvailability-debuginfo                54 kB/s | 1.5 kB     00:00
Apr  6 17:56:03 rdma06 dnf[533272]: beaker-NFV                                       69 kB/s | 2.7 kB     00:00
Apr  6 17:56:03 rdma06 dnf[533272]: beaker-NFV-debuginfo                             52 kB/s | 1.5 kB     00:00
Apr  6 17:56:03 rdma06 dnf[533272]: beaker-RT                                        61 kB/s | 2.7 kB     00:00
Apr  6 17:56:03 rdma06 dnf[533272]: beaker-RT-debuginfo                              42 kB/s | 1.5 kB     00:00
Apr  6 17:56:03 rdma06 dnf[533272]: beaker-ResilientStorage                          68 kB/s | 2.8 kB     00:00
Apr  6 17:56:03 rdma06 dnf[533272]: beaker-ResilientStorage-debuginfo                46 kB/s | 1.5 kB     00:00
Apr  6 17:56:04 rdma06 dnf[533272]: beaker-SAP                                       54 kB/s | 2.7 kB     00:00
Apr  6 17:56:04 rdma06 dnf[533272]: beaker-SAP-debuginfo                             76 kB/s | 1.5 kB     00:00
Apr  6 17:56:04 rdma06 dnf[533272]: beaker-SAPHANA                                   83 kB/s | 2.7 kB     00:00
Apr  6 17:56:04 rdma06 dnf[533272]: beaker-SAPHANA-debuginfo                         38 kB/s | 1.5 kB     00:00
Apr  6 17:56:04 rdma06 dnf[533272]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr  6 17:56:05 rdma06 dnf[533272]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr  6 17:56:05 rdma06 dnf[533272]: Metadata cache created.
Apr  6 17:56:05 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  6 17:56:05 rdma06 systemd[1]: Finished dnf makecache.
Apr  6 17:56:05 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.347s CPU time.
Apr  6 17:58:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 17:58:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 17:58:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 17:58:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.056 sec sweep
Apr  6 18:03:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 18:03:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 18:03:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 18:03:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  6 18:08:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 18:08:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 18:08:22 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 18:08:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 18:13:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 18:13:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 18:13:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 18:13:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 18:18:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 18:18:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 18:18:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 18:18:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  6 18:23:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 18:23:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 18:23:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 18:23:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 18:28:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 18:28:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 18:28:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 18:28:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 18:33:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 18:33:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 18:33:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 18:33:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 18:38:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 18:38:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 18:38:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 18:38:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 18:43:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 18:43:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 18:43:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 18:43:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 18:48:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 18:48:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 18:48:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 18:48:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 18:53:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 18:53:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 18:53:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 18:53:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 18:58:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 18:58:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 18:58:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 18:58:23 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.053 sec sweep
Apr  6 19:03:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 19:03:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 19:03:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 19:03:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 19:08:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 19:08:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 19:08:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 19:08:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 19:13:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 19:13:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 19:13:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 19:13:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 19:18:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 19:18:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 19:18:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 19:18:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 19:23:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 19:23:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 19:23:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 19:23:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 19:28:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 19:28:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 19:28:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 19:28:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 19:33:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 19:33:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 19:33:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 19:33:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 19:38:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 19:38:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 19:38:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 19:38:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 19:43:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 19:43:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 19:43:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 19:43:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 19:46:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  6 19:46:01 rdma06 dnf[537736]: Updating Subscription Management repositories.
Apr  6 19:46:01 rdma06 dnf[537736]: Unable to read consumer identity
Apr  6 19:46:01 rdma06 dnf[537736]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  6 19:46:01 rdma06 dnf[537736]: Metadata cache refreshed recently.
Apr  6 19:46:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  6 19:46:01 rdma06 systemd[1]: Finished dnf makecache.
Apr  6 19:48:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 19:48:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 19:48:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 19:48:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  6 19:53:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 19:53:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 19:53:24 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 19:53:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 19:58:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 19:58:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 19:58:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 19:58:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 20:03:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 20:03:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 20:03:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 20:03:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 20:08:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 20:08:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 20:08:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 20:08:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 20:13:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 20:13:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 20:13:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 20:13:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 20:18:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 20:18:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 20:18:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 20:18:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 20:23:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 20:23:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 20:23:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 20:23:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 20:28:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 20:28:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 20:28:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 20:28:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 20:33:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 20:33:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 20:33:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 20:33:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 20:38:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 20:38:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 20:38:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 20:38:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 20:43:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 20:43:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 20:43:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 20:43:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 20:48:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 20:48:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 20:48:25 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 20:48:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.053 sec sweep
Apr  6 20:48:27 rdma06 chronyd[1034]: Selected source 10.11.160.238
Apr  6 20:50:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  6 20:50:01 rdma06 dnf[540323]: Updating Subscription Management repositories.
Apr  6 20:50:01 rdma06 dnf[540323]: Unable to read consumer identity
Apr  6 20:50:01 rdma06 dnf[540323]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  6 20:50:01 rdma06 dnf[540323]: Metadata cache refreshed recently.
Apr  6 20:50:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  6 20:50:01 rdma06 systemd[1]: Finished dnf makecache.
Apr  6 20:53:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 20:53:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 20:53:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 20:53:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 20:58:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 20:58:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 20:58:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 20:58:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 21:03:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 21:03:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 21:03:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 21:03:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 21:08:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 21:08:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 21:08:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 21:08:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  6 21:13:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 21:13:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 21:13:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 21:13:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  6 21:18:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 21:18:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 21:18:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 21:18:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 21:23:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 21:23:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 21:23:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 21:23:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 21:28:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 21:28:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 21:28:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 21:28:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 21:33:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 21:33:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 21:33:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 21:33:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 21:38:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 21:38:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 21:38:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 21:38:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 21:43:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 21:43:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 21:43:26 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 21:43:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 21:48:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 21:48:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 21:48:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 21:48:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 21:50:50 rdma06 systemd[1]: Starting dnf makecache...
Apr  6 21:50:51 rdma06 dnf[542780]: Updating Subscription Management repositories.
Apr  6 21:50:51 rdma06 dnf[542780]: Unable to read consumer identity
Apr  6 21:50:51 rdma06 dnf[542780]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  6 21:50:52 rdma06 dnf[542780]: Copr repo for fmf owned by psss                 2.8 kB/s | 3.3 kB     00:01
Apr  6 21:50:52 rdma06 dnf[542780]: beaker-AppStream                                 39 kB/s | 3.2 kB     00:00
Apr  6 21:50:52 rdma06 dnf[542780]: beaker-AppStream-debuginfo                       30 kB/s | 1.5 kB     00:00
Apr  6 21:50:53 rdma06 dnf[542780]: beaker-BaseOS                                    48 kB/s | 2.7 kB     00:00
Apr  6 21:50:53 rdma06 dnf[542780]: beaker-BaseOS-debuginfo                          55 kB/s | 1.5 kB     00:00
Apr  6 21:50:53 rdma06 dnf[542780]: beaker-CRB                                       51 kB/s | 2.7 kB     00:00
Apr  6 21:50:53 rdma06 dnf[542780]: beaker-CRB-debuginfo                             68 kB/s | 1.5 kB     00:00
Apr  6 21:50:53 rdma06 dnf[542780]: beaker-HighAvailability                          62 kB/s | 2.8 kB     00:00
Apr  6 21:50:53 rdma06 dnf[542780]: beaker-HighAvailability-debuginfo                52 kB/s | 1.5 kB     00:00
Apr  6 21:50:53 rdma06 dnf[542780]: beaker-NFV                                       71 kB/s | 2.7 kB     00:00
Apr  6 21:50:53 rdma06 dnf[542780]: beaker-NFV-debuginfo                             53 kB/s | 1.5 kB     00:00
Apr  6 21:50:53 rdma06 dnf[542780]: beaker-RT                                        62 kB/s | 2.7 kB     00:00
Apr  6 21:50:53 rdma06 dnf[542780]: beaker-RT-debuginfo                              36 kB/s | 1.5 kB     00:00
Apr  6 21:50:53 rdma06 dnf[542780]: beaker-ResilientStorage                          61 kB/s | 2.8 kB     00:00
Apr  6 21:50:53 rdma06 dnf[542780]: beaker-ResilientStorage-debuginfo                47 kB/s | 1.5 kB     00:00
Apr  6 21:50:53 rdma06 dnf[542780]: beaker-SAP                                      159 kB/s | 2.7 kB     00:00
Apr  6 21:50:53 rdma06 dnf[542780]: beaker-SAP-debuginfo                             59 kB/s | 1.5 kB     00:00
Apr  6 21:50:53 rdma06 dnf[542780]: beaker-SAPHANA                                   96 kB/s | 2.7 kB     00:00
Apr  6 21:50:53 rdma06 dnf[542780]: beaker-SAPHANA-debuginfo                         35 kB/s | 1.5 kB     00:00
Apr  6 21:50:54 rdma06 dnf[542780]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr  6 21:50:55 rdma06 dnf[542780]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr  6 21:50:55 rdma06 dnf[542780]: Metadata cache created.
Apr  6 21:50:55 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  6 21:50:55 rdma06 systemd[1]: Finished dnf makecache.
Apr  6 21:50:55 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.416s CPU time.
Apr  6 21:53:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 21:53:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 21:53:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 21:53:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  6 21:58:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 21:58:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 21:58:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 21:58:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 22:03:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 22:03:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 22:03:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 22:03:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 22:08:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 22:08:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 22:08:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 22:08:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 22:13:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 22:13:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 22:13:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 22:13:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 22:18:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 22:18:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 22:18:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 22:18:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  6 22:20:33 rdma06 NetworkManager[1025]: <info>  [1680834033.0233] dhcp4 (eno1): state changed new lease, address=10.73.131.45
Apr  6 22:23:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 22:23:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 22:23:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 22:23:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  6 22:28:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 22:28:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 22:28:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 22:28:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  6 22:33:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 22:33:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 22:33:27 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 22:33:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 22:35:00 rdma06 systemd[5685]: Starting Cleanup of User's Temporary Files and Directories...
Apr  6 22:35:00 rdma06 systemd[5685]: Finished Cleanup of User's Temporary Files and Directories.
Apr  6 22:38:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 22:38:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 22:38:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 22:38:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 22:41:50 rdma06 systemd[1]: Starting Cleanup of Temporary Directories...
Apr  6 22:41:50 rdma06 systemd[1]: systemd-tmpfiles-clean.service: Deactivated successfully.
Apr  6 22:41:50 rdma06 systemd[1]: Finished Cleanup of Temporary Directories.
Apr  6 22:41:50 rdma06 systemd[1]: run-credentials-systemd\x2dtmpfiles\x2dclean.service.mount: Deactivated successfully.
Apr  6 22:43:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 22:43:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 22:43:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 22:43:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  6 22:48:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 22:48:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 22:48:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 22:48:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  6 22:53:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 22:53:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 22:53:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 22:53:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  6 22:57:38 rdma06 journal[2657]: Adjustment to local watchdog ignored since 'no_localwatchdog' metadata is set
Apr  6 22:58:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 22:58:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 22:58:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 22:58:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  6 23:03:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 23:03:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 23:03:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 23:03:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  6 23:08:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 23:08:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 23:08:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 23:08:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 23:13:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 23:13:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 23:13:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 23:13:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 23:18:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 23:18:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 23:18:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 23:18:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 23:23:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 23:23:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 23:23:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 23:23:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  6 23:28:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 23:28:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 23:28:28 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 23:28:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 23:33:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 23:33:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 23:33:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 23:33:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.056 sec sweep
Apr  6 23:35:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  6 23:35:01 rdma06 dnf[547071]: Updating Subscription Management repositories.
Apr  6 23:35:01 rdma06 dnf[547071]: Unable to read consumer identity
Apr  6 23:35:01 rdma06 dnf[547071]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  6 23:35:01 rdma06 dnf[547071]: Metadata cache refreshed recently.
Apr  6 23:35:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  6 23:35:01 rdma06 systemd[1]: Finished dnf makecache.
Apr  6 23:38:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 23:38:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 23:38:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 23:38:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 23:43:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 23:43:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 23:43:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 23:43:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 23:48:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 23:48:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 23:48:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 23:48:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 23:53:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 23:53:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 23:53:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 23:53:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  6 23:58:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  6 23:58:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  6 23:58:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  6 23:58:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  7 00:01:00 rdma06 systemd[1]: Starting Rotate log files...
Apr  7 00:01:00 rdma06 systemd[1]: logrotate.service: Deactivated successfully.
Apr  7 00:01:00 rdma06 systemd[1]: Finished Rotate log files.
Apr  7 00:03:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  7 00:03:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  7 00:03:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  7 00:03:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  7 00:08:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  7 00:08:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  7 00:08:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  7 00:08:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  7 00:13:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  7 00:13:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  7 00:13:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  7 00:13:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  7 00:18:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  7 00:18:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  7 00:18:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  7 00:18:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  7 00:23:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  7 00:23:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  7 00:23:29 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  7 00:23:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  7 00:28:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  7 00:28:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  7 00:28:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  7 00:28:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  7 00:33:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  7 00:33:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  7 00:33:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  7 00:33:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  7 00:38:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  7 00:38:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  7 00:38:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  7 00:38:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  7 00:43:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  7 00:43:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  7 00:43:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  7 00:43:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  7 00:48:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  7 00:48:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  7 00:48:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  7 00:48:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  7 00:53:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  7 00:53:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  7 00:53:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  7 00:53:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  7 00:54:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  7 00:54:01 rdma06 dnf[550272]: Updating Subscription Management repositories.
Apr  7 00:54:01 rdma06 dnf[550272]: Unable to read consumer identity
Apr  7 00:54:01 rdma06 dnf[550272]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  7 00:54:01 rdma06 dnf[550272]: Copr repo for fmf owned by psss                 4.9 kB/s | 3.3 kB     00:00
Apr  7 00:54:02 rdma06 dnf[550272]: beaker-AppStream                                 48 kB/s | 3.2 kB     00:00
Apr  7 00:54:02 rdma06 dnf[550272]: beaker-AppStream-debuginfo                       37 kB/s | 1.5 kB     00:00
Apr  7 00:54:02 rdma06 dnf[550272]: beaker-BaseOS                                    63 kB/s | 2.7 kB     00:00
Apr  7 00:54:02 rdma06 dnf[550272]: beaker-BaseOS-debuginfo                          47 kB/s | 1.5 kB     00:00
Apr  7 00:54:02 rdma06 dnf[550272]: beaker-CRB                                       52 kB/s | 2.7 kB     00:00
Apr  7 00:54:02 rdma06 dnf[550272]: beaker-CRB-debuginfo                             75 kB/s | 1.5 kB     00:00
Apr  7 00:54:02 rdma06 dnf[550272]: beaker-HighAvailability                          52 kB/s | 2.8 kB     00:00
Apr  7 00:54:02 rdma06 dnf[550272]: beaker-HighAvailability-debuginfo                45 kB/s | 1.5 kB     00:00
Apr  7 00:54:02 rdma06 dnf[550272]: beaker-NFV                                       65 kB/s | 2.7 kB     00:00
Apr  7 00:54:02 rdma06 dnf[550272]: beaker-NFV-debuginfo                             27 kB/s | 1.5 kB     00:00
Apr  7 00:54:02 rdma06 dnf[550272]: beaker-RT                                        59 kB/s | 2.7 kB     00:00
Apr  7 00:54:02 rdma06 dnf[550272]: beaker-RT-debuginfo                              63 kB/s | 1.5 kB     00:00
Apr  7 00:54:02 rdma06 dnf[550272]: beaker-ResilientStorage                          43 kB/s | 2.8 kB     00:00
Apr  7 00:54:02 rdma06 dnf[550272]: beaker-ResilientStorage-debuginfo                35 kB/s | 1.5 kB     00:00
Apr  7 00:54:02 rdma06 dnf[550272]: beaker-SAP                                      182 kB/s | 2.7 kB     00:00
Apr  7 00:54:03 rdma06 dnf[550272]: beaker-SAP-debuginfo                             76 kB/s | 1.5 kB     00:00
Apr  7 00:54:03 rdma06 dnf[550272]: beaker-SAPHANA                                   74 kB/s | 2.7 kB     00:00
Apr  7 00:54:03 rdma06 dnf[550272]: beaker-SAPHANA-debuginfo                         34 kB/s | 1.5 kB     00:00
Apr  7 00:54:03 rdma06 dnf[550272]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr  7 00:54:04 rdma06 dnf[550272]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr  7 00:54:04 rdma06 dnf[550272]: Metadata cache created.
Apr  7 00:54:04 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  7 00:54:04 rdma06 systemd[1]: Finished dnf makecache.
Apr  7 00:54:04 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.281s CPU time.
Apr  7 00:58:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  7 00:58:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  7 00:58:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  7 00:58:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.055 sec sweep
Apr  7 01:03:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  7 01:03:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  7 01:03:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  7 01:03:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  7 01:08:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  7 01:08:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  7 01:08:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  7 01:08:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  7 01:13:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  7 01:13:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  7 01:13:30 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  7 01:13:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  7 01:18:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  7 01:18:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  7 01:18:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  7 01:18:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  7 01:23:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  7 01:23:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  7 01:23:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  7 01:23:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  7 01:28:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  7 01:28:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  7 01:28:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  7 01:28:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  7 01:33:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  7 01:33:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  7 01:33:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  7 01:33:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  7 01:38:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  7 01:38:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  7 01:38:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  7 01:38:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  7 01:43:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  7 01:43:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  7 01:43:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  7 01:43:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  7 01:48:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  7 01:48:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  7 01:48:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  7 01:48:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  7 01:53:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  7 01:53:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  7 01:53:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  7 01:53:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  7 01:58:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  7 01:58:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  7 01:58:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  7 01:58:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  7 02:03:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  7 02:03:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  7 02:03:31 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  7 02:03:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  7 02:08:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  7 02:08:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  7 02:08:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  7 02:08:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  7 02:10:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  7 02:10:01 rdma06 dnf[553377]: Updating Subscription Management repositories.
Apr  7 02:10:01 rdma06 dnf[553377]: Unable to read consumer identity
Apr  7 02:10:01 rdma06 dnf[553377]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  7 02:10:01 rdma06 dnf[553377]: Metadata cache refreshed recently.
Apr  7 02:10:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  7 02:10:01 rdma06 systemd[1]: Finished dnf makecache.
Apr  7 02:13:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  7 02:13:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  7 02:13:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  7 02:13:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  7 02:16:22 rdma06 chronyd[1034]: Selected source 10.13.199.1
Apr  7 02:18:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  7 02:18:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  7 02:18:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  7 02:18:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.056 sec sweep
Apr  7 02:23:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  7 02:23:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  7 02:23:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  7 02:23:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  7 02:28:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  7 02:28:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  7 02:28:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  7 02:28:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  7 02:33:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  7 02:33:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  7 02:33:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  7 02:33:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  7 02:38:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  7 02:38:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  7 02:38:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  7 02:38:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  7 02:43:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  7 02:43:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  7 02:43:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  7 02:43:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  7 02:48:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  7 02:48:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  7 02:48:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  7 02:48:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  7 02:53:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  7 02:53:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  7 02:53:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  7 02:53:32 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  7 02:58:33 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Scheduled sweep interval
Apr  7 02:58:33 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_set_local_port_pkey: sm pkey table already set
Apr  7 02:58:33 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_setup_routing_cost_matrix: Host SM's port is connected to another HFI port
Apr  7 02:58:33 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: DISCOVERY CYCLE END. 0 SWs, 2 HFIs, 2 end ports, 2 total ports, 1 SM(s), 12 packets, 0 retries, 0.054 sec sweep
Apr  7 03:01:38 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: 8051: Link down
Apr  7 03:01:38 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: set_link_state: current ACTIVE, new OFFLINE 
Apr  7 03:01:38 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: 8051: Link down
Apr  7 03:01:38 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: handle_8051_interrupt: link down request already queued
Apr  7 03:01:38 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: 8051: Link down
Apr  7 03:01:38 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: handle_8051_interrupt: link down request already queued
Apr  7 03:01:38 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: 8051: Link down
Apr  7 03:01:38 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: handle_8051_interrupt: link down request already queued
Apr  7 03:01:38 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: physical state changed to PHYS_OFFLINE (0x9), phy 0x95
Apr  7 03:01:39 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: physical state changed to PHYS_OFFLINE (0x9), phy 0x90
Apr  7 03:01:39 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: logical state changed to PORT_DOWN (0x1)
Apr  7 03:01:39 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: Link down reason: Unexpected link down
Apr  7 03:01:39 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: Setting partition keys
Apr  7 03:01:39 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: set_link_state: current OFFLINE, new POLL 
Apr  7 03:01:39 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: Downloading fabric firmware
Apr  7 03:01:39 rdma06 kernel: tg3 0000:03:00.1 eno2: Link is down
Apr  7 03:01:39 rdma06 NetworkManager[1025]: <info>  [1680850899.6966] device (eno2): state change: disconnected -> unavailable (reason 'carrier-changed', sys-iface-state: 'managed')
Apr  7 03:01:39 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: physical state changed to PHYS_POLL (0x2), phy 0x20
Apr  7 03:01:39 rdma06 fm0_sm[380791]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Our SM's port wasn't ready, or went down unexpectedly.
Apr  7 03:01:39 rdma06 fm0_sm[380791]: WARN [topology]: SM: _query_local_port: Waiting for port (portnum=1) to reach state INIT (portstate=Down)...
Apr  7 03:01:39 rdma06 fm0_sm[380791]: PROGR[topology]: SM: sm_transition: SM STATE TRANSITION from MASTER to DISCOVERING
Apr  7 03:01:42 rdma06 kernel: tg3 0000:03:00.1 eno2: Link is up at 100 Mbps, full duplex
Apr  7 03:01:42 rdma06 kernel: tg3 0000:03:00.1 eno2: Flow control is on for TX and on for RX
Apr  7 03:01:42 rdma06 kernel: tg3 0000:03:00.1 eno2: EEE is enabled
Apr  7 03:01:42 rdma06 NetworkManager[1025]: <info>  [1680850902.4609] device (eno2): carrier: link connected
Apr  7 03:01:42 rdma06 NetworkManager[1025]: <info>  [1680850902.4616] device (eno2): state change: unavailable -> disconnected (reason 'carrier-changed', sys-iface-state: 'managed')
Apr  7 03:01:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:01:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:01:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1265 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LD=1 LQI=0
Apr  7 03:01:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:01:47 rdma06 fm0_sm[380791]: WARN [pm]: IF3: if3_check_sa: Port not yet active state: 1
Apr  7 03:01:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:01:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:01:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:01:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:02:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:02:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:02:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:02:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:02:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:02:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:02:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:02:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:02:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:02:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:02:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:02:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:02:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:02:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:02:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:02:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:02:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:02:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:02:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:02:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:02:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:02:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:02:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:02:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:02:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:02:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:02:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:03:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:03:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:03:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:03:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:03:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:03:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:03:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:03:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:03:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:03:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:03:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:03:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:03:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:03:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:03:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:03:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:03:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:03:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:03:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:03:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:03:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:03:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:03:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:03:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:03:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:03:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:03:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:03:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:04:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:04:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:04:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:04:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:04:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:04:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:04:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:04:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:04:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:04:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:04:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:04:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:04:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:04:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:04:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:04:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:04:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:04:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:04:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:04:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:04:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:04:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:04:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:04:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:04:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:04:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:04:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:05:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:05:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:05:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:05:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:05:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:05:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:05:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:05:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:05:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:05:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:05:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:05:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:05:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:05:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:05:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:05:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:05:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:05:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:05:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:05:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:05:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:05:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:05:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:05:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:05:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:05:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:05:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:06:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:06:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:06:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:06:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:06:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:06:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:06:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:06:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:06:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:06:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:06:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:06:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:06:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:06:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:06:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:06:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:06:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:06:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:06:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:06:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:06:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:06:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:06:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:06:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:06:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:06:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:06:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:06:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:07:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:07:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:07:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:07:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:07:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:07:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:07:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:07:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:07:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:07:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:07:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:07:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:07:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:07:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:07:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:07:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:07:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:07:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:07:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:07:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:07:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:07:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:07:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:07:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:07:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:07:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:07:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:08:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:08:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:08:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:08:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:08:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:08:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:08:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:08:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:08:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:08:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:08:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:08:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:08:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:08:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:08:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:08:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:08:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:08:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:08:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:08:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:08:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:08:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:08:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:08:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:08:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:08:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:08:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:09:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:09:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:09:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:09:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:09:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:09:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:09:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:09:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:09:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:09:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:09:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:09:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:09:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:09:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:09:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:09:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:09:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:09:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:09:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:09:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:09:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:09:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:09:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:09:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:09:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:09:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:09:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:09:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:10:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:10:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:10:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:10:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:10:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:10:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:10:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:10:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:10:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:10:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:10:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:10:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:10:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:10:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:10:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:10:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:10:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:10:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:10:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:10:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:10:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:10:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:10:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:10:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:10:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:10:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:10:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:11:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:11:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:11:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:11:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:11:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:11:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:11:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:11:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:11:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:11:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:11:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:11:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:11:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:11:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:11:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:11:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:11:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:11:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:11:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:11:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:11:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:11:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:11:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:11:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:11:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:11:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:11:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:12:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:12:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:12:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:12:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:12:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:12:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:12:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:12:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:12:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:12:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:12:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:12:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:12:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:12:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:12:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:12:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:12:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:12:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:12:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:12:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:12:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:12:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:12:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:12:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:12:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:12:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:12:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:12:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:13:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:13:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:13:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:13:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:13:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:13:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:13:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:13:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:13:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:13:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:13:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:13:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:13:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:13:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:13:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:13:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:13:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:13:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:13:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:13:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:13:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:13:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:13:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:13:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:13:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:13:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:13:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:14:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:14:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:14:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:14:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:14:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:14:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:14:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:14:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:14:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:14:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:14:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:14:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:14:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:14:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:14:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:14:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:14:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:14:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:14:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:14:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:14:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:14:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:14:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:14:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:14:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:14:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:14:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:15:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:15:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:15:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:15:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:15:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:15:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:15:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:15:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:15:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:15:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:15:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:15:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:15:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:15:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:15:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:15:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:15:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:15:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:15:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:15:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:15:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:15:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:15:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:15:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:15:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:15:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:15:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:15:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:16:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:16:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:16:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:16:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:16:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:16:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:16:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:16:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:16:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:16:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:16:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:16:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:16:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:16:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:16:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:16:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:16:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:16:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:16:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:16:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:16:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:16:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:16:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:16:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:16:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:16:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:16:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:17:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:17:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:17:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:17:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:17:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:17:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:17:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:17:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:17:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:17:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:17:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:17:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:17:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:17:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:17:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:17:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:17:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:17:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:17:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:17:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:17:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:17:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:17:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:17:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:17:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:17:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:17:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:18:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:18:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:18:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:18:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:18:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:18:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:18:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:18:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:18:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:18:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:18:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:18:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:18:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:18:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:18:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:18:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:18:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:18:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:18:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:18:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:18:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:18:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:18:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:18:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:18:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:18:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:18:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:19:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:19:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:19:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:19:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:19:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:19:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:19:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:19:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:19:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:19:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:19:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:19:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:19:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:19:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:19:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:19:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:19:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:19:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:19:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:19:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:19:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:19:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:19:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:19:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:19:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:19:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:19:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:19:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:20:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:20:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:20:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:20:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:20:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:20:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:20:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:20:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:20:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:20:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:20:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:20:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:20:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:20:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:20:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:20:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:20:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:20:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:20:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:20:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:20:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:20:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:20:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:20:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:20:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:20:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:20:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:21:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:21:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:21:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:21:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:21:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:21:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:21:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:21:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:21:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:21:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:21:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:21:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:21:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:21:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:21:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:21:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:21:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:21:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:21:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:21:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:21:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:21:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:21:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:21:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:21:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:21:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:21:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:22:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:22:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:22:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:22:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:22:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:22:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:22:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:22:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:22:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:22:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:22:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:22:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:22:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:22:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:22:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:22:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:22:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:22:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:22:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:22:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:22:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:22:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:22:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:22:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:22:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:22:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:22:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:22:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:23:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:23:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:23:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:23:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:23:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:23:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:23:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:23:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:23:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:23:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:23:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:23:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:23:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:23:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:23:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:23:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:23:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:23:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:23:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:23:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:23:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:23:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:23:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:23:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:23:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:23:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:23:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:24:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:24:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:24:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:24:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:24:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:24:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:24:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:24:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:24:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:24:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:24:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:24:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:24:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:24:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:24:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:24:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:24:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:24:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:24:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:24:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:24:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:24:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:24:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:24:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:24:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:24:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:24:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:25:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:25:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:25:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:25:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:25:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:25:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:25:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:25:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:25:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:25:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:25:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:25:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:25:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:25:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:25:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:25:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:25:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:25:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:25:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:25:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:25:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:25:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:25:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:25:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:25:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:25:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:25:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:25:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:26:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:26:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:26:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:26:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:26:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:26:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:26:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:26:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:26:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:26:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:26:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:26:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:26:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:26:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:26:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:26:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:26:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:26:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:26:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:26:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:26:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:26:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:26:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:26:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:26:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:26:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:26:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:27:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:27:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:27:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:27:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:27:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:27:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:27:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:27:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:27:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:27:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:27:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:27:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:27:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:27:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:27:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:27:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:27:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:27:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:27:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:27:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:27:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:27:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:27:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:27:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:27:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:27:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:27:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:28:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:28:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:28:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:28:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:28:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:28:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:28:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:28:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:28:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:28:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:28:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:28:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:28:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:28:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:28:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:28:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:28:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:28:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:28:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:28:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:28:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:28:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:28:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:28:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:28:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:28:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:28:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:28:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:29:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:29:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:29:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:29:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:29:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:29:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:29:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:29:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:29:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:29:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:29:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:29:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:29:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:29:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:29:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:29:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:29:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:29:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:29:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:29:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:29:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:29:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:29:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:29:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:29:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:29:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:29:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:30:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:30:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:30:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:30:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:30:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:30:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:30:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:30:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:30:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:30:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:30:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:30:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:30:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:30:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:30:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:30:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:30:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:30:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:30:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:30:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:30:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:30:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:30:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:30:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:30:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:30:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:30:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:31:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:31:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:31:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:31:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:31:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:31:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:31:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:31:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:31:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:31:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:31:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:31:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:31:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:31:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:31:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:31:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:31:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:31:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:31:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:31:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:31:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:31:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:31:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:31:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:31:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:31:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:31:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:31:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:32:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:32:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:32:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:32:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:32:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:32:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:32:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:32:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:32:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:32:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:32:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:32:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:32:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:32:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:32:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:32:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:32:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:32:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:32:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:32:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:32:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:32:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:32:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:32:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:32:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:32:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:32:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:33:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:33:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:33:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:33:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:33:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:33:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:33:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:33:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:33:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:33:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:33:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:33:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:33:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:33:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:33:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:33:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:33:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:33:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:33:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:33:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:33:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:33:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:33:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:33:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:33:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:33:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:33:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:34:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:34:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:34:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:34:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:34:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:34:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:34:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:34:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:34:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:34:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:34:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:34:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:34:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:34:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:34:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:34:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:34:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:34:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:34:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:34:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:34:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:34:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:34:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:34:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:34:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:34:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:34:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:34:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:35:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:35:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:35:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:35:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:35:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:35:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:35:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:35:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:35:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:35:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:35:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:35:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:35:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:35:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:35:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:35:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:35:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:35:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:35:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:35:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:35:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:35:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:35:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:35:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:35:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:35:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:35:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:36:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:36:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:36:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:36:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:36:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:36:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:36:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:36:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:36:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:36:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:36:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:36:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:36:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:36:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:36:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:36:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:36:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:36:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:36:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:36:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:36:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:36:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:36:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:36:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:36:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:36:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:36:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:37:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:37:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:37:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:37:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:37:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:37:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:37:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:37:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:37:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:37:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:37:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:37:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:37:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:37:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:37:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:37:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:37:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:37:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:37:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:37:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:37:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:37:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:37:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:37:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:37:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:37:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:37:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:38:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  7 03:38:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:38:01 rdma06 dnf[556938]: Updating Subscription Management repositories.
Apr  7 03:38:01 rdma06 dnf[556938]: Unable to read consumer identity
Apr  7 03:38:01 rdma06 dnf[556938]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  7 03:38:01 rdma06 dnf[556938]: Metadata cache refreshed recently.
Apr  7 03:38:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  7 03:38:01 rdma06 systemd[1]: Finished dnf makecache.
Apr  7 03:38:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:38:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:38:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:38:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:38:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:38:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:38:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:38:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:38:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:38:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:38:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:38:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:38:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:38:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:38:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:38:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:38:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:38:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:38:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:38:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:38:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:38:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:38:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:38:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:38:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:38:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:38:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:39:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:39:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:39:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:39:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:39:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:39:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:39:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:39:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:39:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:39:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:39:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:39:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:39:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:39:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:39:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:39:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:39:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:39:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:39:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:39:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:39:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:39:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:39:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:39:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:39:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:39:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:39:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:40:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:40:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:40:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:40:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:40:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:40:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:40:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:40:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:40:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:40:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:40:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:40:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:40:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:40:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:40:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:40:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:40:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:40:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:40:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:40:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:40:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:40:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:40:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:40:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:40:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:40:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:40:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:41:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:41:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:41:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:41:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:41:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:41:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:41:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:41:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:41:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:41:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:41:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:41:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:41:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:41:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:41:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:41:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:41:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:41:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:41:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:41:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:41:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:41:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:41:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:41:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:41:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:41:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:41:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:41:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:42:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:42:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:42:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:42:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:42:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:42:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:42:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:42:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:42:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:42:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:42:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:42:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:42:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:42:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:42:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:42:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:42:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:42:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:42:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:42:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:42:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:42:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:42:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:42:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:42:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:42:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:42:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:43:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:43:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:43:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:43:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:43:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:43:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:43:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:43:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:43:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:43:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:43:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:43:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:43:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:43:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:43:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:43:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:43:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:43:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:43:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:43:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:43:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:43:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:43:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:43:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:43:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:43:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:43:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:44:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:44:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:44:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:44:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:44:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:44:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:44:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:44:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:44:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:44:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:44:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:44:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:44:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:44:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:44:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:44:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:44:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:44:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:44:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:44:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:44:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:44:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:44:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:44:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:44:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:44:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:44:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:44:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:45:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:45:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:45:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:45:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:45:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:45:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:45:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:45:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:45:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:45:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:45:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:45:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:45:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:45:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:45:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:45:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:45:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:45:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:45:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:45:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:45:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:45:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:45:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:45:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:45:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:45:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:45:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:46:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:46:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:46:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:46:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:46:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:46:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:46:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:46:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:46:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:46:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:46:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:46:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:46:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:46:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:46:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:46:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:46:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:46:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:46:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:46:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:46:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:46:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:46:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:46:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:46:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:46:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:46:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:47:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:47:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:47:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:47:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:47:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:47:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:47:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:47:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:47:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:47:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:47:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:47:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:47:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:47:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:47:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:47:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:47:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:47:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:47:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:47:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:47:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:47:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:47:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:47:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:47:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:47:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:47:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:47:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:48:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:48:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:48:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:48:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:48:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:48:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:48:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:48:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:48:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:48:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:48:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:48:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:48:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:48:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:48:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:48:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:48:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:48:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:48:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:48:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:48:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:48:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:48:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:48:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:48:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:48:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:48:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:49:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:49:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:49:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:49:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:49:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:49:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:49:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:49:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:49:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:49:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:49:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:49:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:49:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:49:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:49:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:49:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:49:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:49:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:49:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:49:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:49:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:49:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:49:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:49:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:49:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:49:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:49:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:50:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:50:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:50:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:50:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:50:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:50:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:50:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:50:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:50:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:50:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:50:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:50:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:50:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:50:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:50:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:50:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:50:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:50:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:50:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:50:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:50:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:50:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:50:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:50:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:50:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:50:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:50:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:50:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:51:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:51:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:51:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:51:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:51:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:51:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:51:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:51:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:51:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:51:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:51:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:51:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:51:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:51:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:51:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:51:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:51:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:51:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:51:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:51:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:51:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:51:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:51:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:51:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:51:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:51:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:51:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:52:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:52:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:52:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:52:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:52:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:52:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:52:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:52:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:52:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:52:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:52:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:52:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:52:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:52:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:52:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:52:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:52:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:52:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:52:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:52:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:52:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:52:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:52:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:52:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:52:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:52:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:52:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:53:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:53:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:53:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:53:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:53:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:53:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:53:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:53:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:53:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:53:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:53:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:53:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:53:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:53:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:53:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:53:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:53:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:53:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:53:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:53:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:53:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:53:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:53:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:53:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:53:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:53:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:53:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:54:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:54:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:54:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:54:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:54:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:54:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:54:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:54:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:54:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:54:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:54:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:54:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:54:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:54:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:54:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:54:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:54:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:54:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:54:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:54:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:54:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:54:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:54:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:54:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:54:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:54:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:54:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:54:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:55:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:55:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:55:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:55:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:55:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:55:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:55:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:55:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:55:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:55:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:55:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:55:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:55:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:55:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:55:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:55:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:55:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:55:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:55:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:55:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:55:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:55:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:55:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:55:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:55:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:55:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:55:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:56:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:56:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:56:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:56:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:56:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:56:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:56:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:56:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:56:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:56:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:56:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:56:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:56:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:56:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:56:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:56:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:56:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:56:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:56:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:56:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:56:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:56:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:56:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:56:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:56:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:56:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:56:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:57:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:57:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:57:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:57:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:57:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:57:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:57:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:57:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:57:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:57:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:57:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:57:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:57:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:57:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:57:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:57:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:57:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:57:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:57:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:57:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:57:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:57:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:57:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:57:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:57:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:57:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:57:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:57:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:58:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:58:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:58:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:58:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:58:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:58:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:58:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:58:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:58:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:58:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:58:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:58:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:58:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:58:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:58:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:58:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:58:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:58:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:58:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:58:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:58:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:58:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:58:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:58:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:58:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:58:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:58:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:59:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:59:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:59:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:59:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:59:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:59:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:59:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:59:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:59:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:59:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:59:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:59:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:59:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:59:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:59:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:59:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:59:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:59:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:59:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 03:59:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:59:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:59:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:59:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 03:59:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 03:59:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 03:59:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 03:59:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:00:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:00:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:00:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:00:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:00:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:00:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:00:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:00:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:00:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:00:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:00:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:00:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:00:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:00:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:00:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:00:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:00:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:00:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:00:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:00:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:00:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:00:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:00:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:00:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:00:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:00:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:00:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:00:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:01:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:01:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:01:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:01:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:01:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:01:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:01:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:01:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:01:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:01:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:01:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:01:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:01:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:01:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:01:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:01:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:01:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:01:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:01:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:01:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:01:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:01:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:01:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:01:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:01:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:01:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:01:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:02:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:02:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:02:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:02:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:02:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:02:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:02:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:02:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:02:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:02:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:02:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:02:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:02:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:02:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:02:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:02:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:02:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:02:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:02:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:02:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:02:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:02:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:02:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:02:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:02:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:02:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:02:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:03:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:03:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:03:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:03:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:03:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:03:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:03:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:03:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:03:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:03:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:03:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:03:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:03:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:03:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:03:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:03:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:03:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:03:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:03:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:03:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:03:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:03:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:03:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:03:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:03:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:03:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:03:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:03:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:04:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:04:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:04:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:04:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:04:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:04:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:04:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:04:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:04:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:04:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:04:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:04:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:04:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:04:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:04:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:04:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:04:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:04:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:04:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:04:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:04:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:04:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:04:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:04:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:04:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:04:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:04:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:05:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:05:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:05:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:05:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:05:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:05:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:05:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:05:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:05:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:05:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:05:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:05:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:05:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:05:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:05:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:05:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:05:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:05:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:05:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:05:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:05:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:05:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:05:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:05:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:05:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:05:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:05:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:06:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:06:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:06:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:06:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:06:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:06:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:06:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:06:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:06:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:06:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:06:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:06:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:06:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:06:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:06:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:06:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:06:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:06:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:06:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:06:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:06:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:06:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:06:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:06:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:06:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:06:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:06:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:06:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:07:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:07:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:07:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:07:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:07:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:07:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:07:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:07:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:07:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:07:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:07:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:07:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:07:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:07:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:07:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:07:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:07:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:07:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:07:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:07:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:07:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:07:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:07:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:07:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:07:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:07:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:07:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:08:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:08:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:08:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:08:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:08:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:08:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:08:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:08:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:08:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:08:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:08:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:08:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:08:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:08:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:08:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:08:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:08:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:08:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:08:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:08:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:08:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:08:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:08:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:08:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:08:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:08:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:08:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:09:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:09:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:09:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:09:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:09:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:09:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:09:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:09:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:09:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:09:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:09:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:09:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:09:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:09:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:09:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:09:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:09:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:09:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:09:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:09:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:09:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:09:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:09:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:09:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:09:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:09:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:09:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:10:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:10:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:10:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:10:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:10:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:10:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:10:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:10:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:10:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:10:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:10:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:10:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:10:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:10:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:10:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:10:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:10:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:10:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:10:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:10:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:10:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:10:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:10:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:10:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:10:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:10:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:10:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:10:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:11:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:11:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:11:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:11:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:11:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:11:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:11:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:11:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:11:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:11:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:11:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:11:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:11:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:11:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:11:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:11:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:11:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:11:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:11:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:11:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:11:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:11:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:11:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:11:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:11:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:11:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:11:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:12:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:12:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:12:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:12:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:12:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:12:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:12:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:12:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:12:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:12:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:12:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:12:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:12:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:12:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:12:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:12:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:12:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:12:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:12:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:12:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:12:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:12:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:12:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:12:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:12:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:12:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:12:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:13:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:13:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:13:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:13:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:13:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:13:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:13:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:13:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:13:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:13:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:13:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:13:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:13:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:13:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:13:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:13:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:13:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:13:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:13:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:13:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:13:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:13:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:13:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:13:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:13:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:13:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:13:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:13:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:14:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:14:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:14:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:14:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:14:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:14:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:14:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:14:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:14:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:14:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:14:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:14:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:14:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:14:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:14:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:14:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:14:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:14:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:14:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:14:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:14:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:14:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:14:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:14:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:14:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:14:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:14:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:15:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:15:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:15:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:15:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:15:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:15:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:15:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:15:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:15:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:15:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:15:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:15:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:15:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:15:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:15:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:15:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:15:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:15:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:15:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:15:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:15:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:15:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:15:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:15:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:15:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:15:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:15:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:16:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:16:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:16:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:16:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:16:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:16:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:16:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:16:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:16:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:16:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:16:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:16:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:16:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:16:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:16:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:16:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:16:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:16:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:16:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:16:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:16:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:16:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:16:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:16:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:16:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:16:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:16:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:16:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:17:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:17:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:17:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:17:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:17:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:17:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:17:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:17:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:17:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:17:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:17:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:17:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:17:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:17:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:17:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:17:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:17:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:17:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:17:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:17:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:17:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:17:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:17:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:17:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:17:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:17:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:17:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:18:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:18:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:18:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:18:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:18:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:18:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:18:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:18:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:18:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:18:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:18:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:18:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:18:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:18:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:18:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:18:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:18:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:18:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:18:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:18:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:18:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:18:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:18:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:18:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:18:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:18:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:18:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:19:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:19:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:19:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:19:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:19:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:19:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:19:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:19:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:19:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:19:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:19:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:19:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:19:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:19:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:19:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:19:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:19:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:19:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:19:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:19:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:19:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:19:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:19:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:19:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:19:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:19:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:19:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:19:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:20:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:20:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:20:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:20:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:20:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:20:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:20:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:20:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:20:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:20:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:20:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:20:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:20:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:20:32 rdma06 NetworkManager[1025]: <info>  [1680855632.9323] dhcp4 (eno1): state changed new lease, address=10.73.131.45
Apr  7 04:20:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:20:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:20:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:20:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:20:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:20:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:20:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:20:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:20:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:20:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:20:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:20:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:20:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:20:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:21:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:21:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:21:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:21:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:21:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:21:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:21:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:21:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:21:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:21:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:21:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:21:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:21:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:21:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:21:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:21:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:21:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:21:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:21:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:21:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:21:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:21:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:21:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:21:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:21:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:21:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:21:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:22:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:22:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:22:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:22:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:22:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:22:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:22:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:22:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:22:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:22:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:22:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:22:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:22:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:22:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:22:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:22:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:22:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:22:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:22:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:22:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:22:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:22:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:22:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:22:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:22:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:22:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:22:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:22:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:23:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:23:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:23:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:23:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:23:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:23:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:23:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:23:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:23:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:23:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:23:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:23:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:23:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:23:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:23:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:23:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:23:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:23:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:23:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:23:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:23:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:23:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:23:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:23:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:23:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:23:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:23:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:24:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:24:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:24:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:24:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:24:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:24:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:24:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:24:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:24:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:24:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:24:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:24:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:24:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:24:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:24:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:24:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:24:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:24:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:24:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:24:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:24:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:24:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:24:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:24:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:24:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:24:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:24:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:25:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:25:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:25:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:25:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:25:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:25:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:25:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:25:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:25:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:25:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:25:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:25:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:25:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:25:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:25:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:25:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:25:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:25:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:25:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:25:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:25:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:25:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:25:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:25:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:25:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:25:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:25:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:26:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:26:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:26:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:26:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:26:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:26:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:26:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:26:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:26:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:26:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:26:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:26:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:26:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:26:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:26:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:26:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:26:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:26:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:26:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:26:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:26:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:26:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:26:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:26:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:26:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:26:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:26:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:26:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:27:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:27:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:27:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:27:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:27:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:27:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:27:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:27:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:27:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:27:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:27:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:27:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:27:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:27:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:27:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:27:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:27:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:27:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:27:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:27:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:27:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:27:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:27:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:27:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:27:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:27:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:27:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:28:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:28:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:28:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:28:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:28:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:28:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:28:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:28:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:28:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:28:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:28:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:28:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:28:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:28:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:28:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:28:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:28:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:28:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:28:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:28:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:28:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:28:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:28:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:28:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:28:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:28:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:28:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:29:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:29:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:29:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:29:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:29:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:29:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:29:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:29:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:29:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:29:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:29:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:29:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:29:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:29:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:29:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:29:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:29:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:29:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:29:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:29:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:29:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:29:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:29:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:29:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:29:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:29:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:29:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:29:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:30:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:30:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:30:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:30:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:30:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:30:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:30:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:30:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:30:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:30:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:30:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:30:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:30:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:30:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:30:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:30:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:30:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:30:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:30:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:30:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:30:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:30:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:30:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:30:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:30:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:30:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:30:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:31:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:31:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:31:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:31:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:31:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:31:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:31:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:31:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:31:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:31:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:31:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:31:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:31:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:31:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:31:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:31:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:31:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:31:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:31:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:31:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:31:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:31:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:31:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:31:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:31:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:31:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:31:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:32:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:32:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:32:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:32:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:32:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:32:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:32:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:32:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:32:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:32:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:32:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:32:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:32:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:32:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:32:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:32:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:32:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:32:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:32:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:32:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:32:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:32:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:32:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:32:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:32:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:32:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:32:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:32:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:33:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:33:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:33:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:33:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:33:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:33:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:33:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:33:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:33:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:33:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:33:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:33:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:33:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:33:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:33:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:33:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:33:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:33:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:33:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:33:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:33:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:33:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:33:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:33:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:33:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:33:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:33:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:34:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:34:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:34:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:34:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:34:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:34:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:34:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:34:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:34:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:34:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:34:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:34:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:34:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:34:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:34:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:34:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:34:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:34:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:34:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:34:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:34:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:34:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:34:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:34:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:34:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:34:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:34:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:35:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:35:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:35:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:35:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:35:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:35:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:35:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:35:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:35:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:35:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:35:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:35:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:35:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:35:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:35:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:35:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:35:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:35:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:35:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:35:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:35:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:35:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:35:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:35:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:35:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:35:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:35:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:35:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:36:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:36:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:36:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:36:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:36:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:36:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:36:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:36:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:36:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:36:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:36:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:36:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:36:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:36:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:36:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:36:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:36:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:36:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:36:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:36:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:36:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:36:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:36:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:36:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:36:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:36:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:36:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:37:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:37:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:37:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:37:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:37:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:37:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:37:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:37:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:37:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:37:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:37:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:37:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:37:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:37:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:37:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:37:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:37:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:37:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:37:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:37:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:37:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:37:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:37:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:37:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:37:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:37:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:37:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:38:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:38:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:38:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:38:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:38:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:38:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:38:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:38:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:38:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:38:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:38:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:38:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:38:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:38:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:38:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:38:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:38:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:38:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:38:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:38:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:38:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:38:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:38:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:38:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:38:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:38:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:38:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:38:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:39:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:39:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:39:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:39:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:39:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:39:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:39:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:39:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:39:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:39:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:39:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:39:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:39:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:39:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:39:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:39:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:39:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:39:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:39:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:39:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:39:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:39:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:39:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:39:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:39:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:39:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:39:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:40:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:40:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:40:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:40:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:40:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:40:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:40:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:40:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:40:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:40:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:40:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:40:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:40:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:40:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:40:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:40:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:40:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:40:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:40:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:40:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:40:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:40:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:40:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:40:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:40:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:40:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:40:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:41:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:41:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:41:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:41:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:41:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:41:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:41:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:41:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:41:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:41:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:41:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:41:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:41:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:41:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:41:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:41:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:41:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:41:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:41:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:41:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:41:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:41:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:41:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:41:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:41:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:41:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:41:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:41:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:42:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:42:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:42:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:42:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:42:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:42:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:42:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:42:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:42:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:42:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:42:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:42:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:42:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:42:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:42:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:42:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:42:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:42:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:42:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:42:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:42:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:42:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:42:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:42:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:42:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:42:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:42:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:43:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:43:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:43:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:43:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:43:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:43:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:43:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:43:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:43:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:43:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:43:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:43:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:43:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:43:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:43:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:43:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:43:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:43:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:43:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:43:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:43:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:43:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:43:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:43:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:43:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:43:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:43:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:44:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:44:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:44:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:44:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:44:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:44:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:44:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:44:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:44:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:44:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:44:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:44:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:44:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:44:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:44:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:44:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:44:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:44:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:44:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:44:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:44:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:44:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:44:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:44:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:44:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:44:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:44:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:45:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:45:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:45:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:45:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:45:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:45:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:45:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:45:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:45:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:45:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:45:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:45:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:45:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:45:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:45:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:45:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:45:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:45:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:45:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:45:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:45:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:45:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:45:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:45:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:45:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:45:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:45:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:45:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:46:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:46:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:46:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:46:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:46:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:46:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:46:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:46:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:46:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:46:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:46:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:46:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:46:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:46:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:46:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:46:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:46:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:46:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:46:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:46:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:46:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:46:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:46:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:46:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:46:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:46:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:46:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:47:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:47:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:47:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:47:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:47:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:47:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:47:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:47:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:47:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:47:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:47:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:47:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:47:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:47:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:47:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:47:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:47:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:47:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:47:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:47:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:47:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:47:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:47:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:47:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:47:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:47:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:47:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:48:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:48:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:48:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:48:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:48:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:48:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:48:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:48:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:48:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:48:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:48:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:48:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:48:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:48:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:48:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:48:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:48:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:48:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:48:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:48:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:48:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:48:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:48:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:48:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:48:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:48:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:48:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:48:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:49:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:49:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:49:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:49:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:49:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:49:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:49:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:49:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:49:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:49:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:49:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:49:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:49:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:49:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:49:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:49:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:49:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:49:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:49:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:49:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:49:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:49:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:49:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:49:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:49:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:49:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:49:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:50:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:50:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:50:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:50:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:50:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:50:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:50:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:50:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:50:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:50:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:50:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:50:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:50:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:50:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:50:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:50:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:50:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:50:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:50:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:50:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:50:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:50:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:50:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:50:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:50:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:50:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:50:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:51:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:51:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:51:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:51:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:51:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:51:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:51:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:51:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:51:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:51:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:51:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:51:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:51:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:51:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:51:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:51:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:51:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:51:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:51:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:51:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:51:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:51:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:51:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:51:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:51:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:51:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:51:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:51:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:52:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:52:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:52:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:52:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:52:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:52:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:52:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:52:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:52:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:52:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:52:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:52:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:52:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:52:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:52:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:52:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:52:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:52:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:52:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:52:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:52:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:52:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:52:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:52:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:52:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:52:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:52:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:53:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:53:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:53:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:53:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:53:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:53:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:53:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:53:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:53:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:53:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:53:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:53:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:53:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:53:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:53:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:53:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:53:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:53:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:53:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:53:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:53:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:53:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:53:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:53:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:53:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:53:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:53:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:54:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  7 04:54:01 rdma06 dnf[560024]: Updating Subscription Management repositories.
Apr  7 04:54:01 rdma06 dnf[560024]: Unable to read consumer identity
Apr  7 04:54:01 rdma06 dnf[560024]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  7 04:54:03 rdma06 dnf[560024]: Copr repo for fmf owned by psss                 2.1 kB/s | 3.3 kB     00:01
Apr  7 04:54:03 rdma06 dnf[560024]: beaker-AppStream                                 86 kB/s | 3.2 kB     00:00
Apr  7 04:54:03 rdma06 dnf[560024]: beaker-AppStream-debuginfo                       34 kB/s | 1.5 kB     00:00
Apr  7 04:54:03 rdma06 dnf[560024]: beaker-BaseOS                                   120 kB/s | 2.7 kB     00:00
Apr  7 04:54:03 rdma06 dnf[560024]: beaker-BaseOS-debuginfo                          25 kB/s | 1.5 kB     00:00
Apr  7 04:54:03 rdma06 dnf[560024]: beaker-CRB                                       55 kB/s | 2.7 kB     00:00
Apr  7 04:54:03 rdma06 dnf[560024]: beaker-CRB-debuginfo                             65 kB/s | 1.5 kB     00:00
Apr  7 04:54:03 rdma06 dnf[560024]: beaker-HighAvailability                          68 kB/s | 2.8 kB     00:00
Apr  7 04:54:03 rdma06 dnf[560024]: beaker-HighAvailability-debuginfo                41 kB/s | 1.5 kB     00:00
Apr  7 04:54:03 rdma06 dnf[560024]: beaker-NFV                                       86 kB/s | 2.7 kB     00:00
Apr  7 04:54:03 rdma06 dnf[560024]: beaker-NFV-debuginfo                             72 kB/s | 1.5 kB     00:00
Apr  7 04:54:03 rdma06 dnf[560024]: beaker-RT                                        66 kB/s | 2.7 kB     00:00
Apr  7 04:54:03 rdma06 dnf[560024]: beaker-RT-debuginfo                              50 kB/s | 1.5 kB     00:00
Apr  7 04:54:03 rdma06 dnf[560024]: beaker-ResilientStorage                          72 kB/s | 2.8 kB     00:00
Apr  7 04:54:03 rdma06 dnf[560024]: beaker-ResilientStorage-debuginfo                54 kB/s | 1.5 kB     00:00
Apr  7 04:54:03 rdma06 dnf[560024]: beaker-SAP                                      136 kB/s | 2.7 kB     00:00
Apr  7 04:54:03 rdma06 dnf[560024]: beaker-SAP-debuginfo                             59 kB/s | 1.5 kB     00:00
Apr  7 04:54:04 rdma06 dnf[560024]: beaker-SAPHANA                                   78 kB/s | 2.7 kB     00:00
Apr  7 04:54:04 rdma06 dnf[560024]: beaker-SAPHANA-debuginfo                         37 kB/s | 1.5 kB     00:00
Apr  7 04:54:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:54:04 rdma06 dnf[560024]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr  7 04:54:05 rdma06 dnf[560024]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr  7 04:54:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:54:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:54:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:54:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:54:05 rdma06 dnf[560024]: Metadata cache created.
Apr  7 04:54:05 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  7 04:54:05 rdma06 systemd[1]: Finished dnf makecache.
Apr  7 04:54:05 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.431s CPU time.
Apr  7 04:54:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:54:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:54:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:54:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:54:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:54:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:54:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:54:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:54:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:54:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:54:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:54:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:54:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:54:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:54:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:54:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:54:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:54:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:54:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:54:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:54:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:54:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:54:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:55:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:55:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:55:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:55:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:55:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:55:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:55:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:55:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:55:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:55:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:55:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:55:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:55:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:55:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:55:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:55:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:55:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:55:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:55:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:55:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:55:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:55:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:55:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:55:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:55:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:55:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:55:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:56:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:56:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:56:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:56:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:56:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:56:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:56:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:56:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:56:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:56:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:56:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:56:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:56:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:56:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:56:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:56:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:56:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:56:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:56:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:56:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:56:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:56:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:56:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:56:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:56:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:56:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:56:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:57:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:57:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:57:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:57:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:57:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:57:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:57:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:57:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:57:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:57:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:57:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:57:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:57:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:57:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:57:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:57:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:57:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:57:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:57:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:57:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:57:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:57:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:57:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:57:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:57:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:57:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:57:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:57:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:58:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:58:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:58:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:58:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:58:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:58:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:58:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:58:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:58:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:58:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:58:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:58:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:58:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:58:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:58:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:58:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:58:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:58:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:58:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:58:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:58:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:58:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:58:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:58:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:58:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:58:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:58:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:59:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:59:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:59:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:59:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:59:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:59:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:59:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:59:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:59:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:59:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:59:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:59:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:59:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:59:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:59:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:59:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:59:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:59:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:59:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:59:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:59:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:59:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 04:59:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 04:59:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 04:59:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 04:59:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 04:59:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:00:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:00:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:00:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:00:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:00:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:00:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:00:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:00:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:00:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:00:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:00:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:00:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:00:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:00:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:00:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:00:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:00:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:00:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:00:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:00:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:00:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:00:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:00:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:00:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:00:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:00:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:00:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:01:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:01:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:01:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:01:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:01:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:01:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:01:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:01:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:01:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:01:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:01:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:01:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:01:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:01:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:01:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:01:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:01:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:01:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:01:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:01:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:01:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:01:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:01:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:01:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:01:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:01:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:01:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:01:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:02:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:02:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:02:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:02:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:02:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:02:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:02:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:02:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:02:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:02:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:02:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:02:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:02:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:02:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:02:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:02:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:02:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:02:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:02:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:02:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:02:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:02:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:02:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:02:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:02:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:02:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:02:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:03:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:03:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:03:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:03:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:03:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:03:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:03:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:03:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:03:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:03:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:03:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:03:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:03:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:03:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:03:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:03:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:03:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:03:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:03:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:03:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:03:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:03:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:03:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:03:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:03:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:03:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:03:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:04:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:04:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:04:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:04:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:04:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:04:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:04:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:04:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:04:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:04:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:04:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:04:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:04:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:04:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:04:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:04:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:04:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:04:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:04:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:04:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:04:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:04:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:04:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:04:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:04:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:04:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:04:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:04:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:05:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:05:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:05:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:05:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:05:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:05:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:05:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:05:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:05:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:05:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:05:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:05:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:05:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:05:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:05:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:05:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:05:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:05:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:05:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:05:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:05:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:05:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:05:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:05:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:05:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:05:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:05:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:06:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:06:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:06:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:06:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:06:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:06:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:06:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:06:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:06:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:06:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:06:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:06:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:06:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:06:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:06:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:06:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:06:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:06:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:06:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:06:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:06:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:06:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:06:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:06:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:06:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:06:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:06:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:07:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:07:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:07:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:07:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:07:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:07:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:07:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:07:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:07:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:07:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:07:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:07:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:07:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:07:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:07:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:07:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:07:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:07:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:07:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:07:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:07:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:07:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:07:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:07:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:07:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:07:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:07:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:07:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:08:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:08:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:08:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:08:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:08:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:08:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:08:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:08:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:08:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:08:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:08:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:08:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:08:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:08:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:08:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:08:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:08:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:08:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:08:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:08:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:08:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:08:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:08:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:08:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:08:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:08:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:08:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:09:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:09:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:09:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:09:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:09:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:09:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:09:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:09:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:09:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:09:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:09:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:09:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:09:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:09:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:09:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:09:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:09:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:09:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:09:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:09:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:09:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:09:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:09:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:09:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:09:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:09:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:09:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:10:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:10:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:10:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:10:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:10:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:10:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:10:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:10:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:10:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:10:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:10:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:10:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:10:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:10:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:10:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:10:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:10:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:10:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:10:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:10:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:10:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:10:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:10:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:10:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:10:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:10:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:10:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:10:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:11:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:11:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:11:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:11:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:11:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:11:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:11:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:11:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:11:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:11:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:11:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:11:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:11:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:11:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:11:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:11:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:11:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:11:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:11:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:11:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:11:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:11:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:11:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:11:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:11:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:11:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:11:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:12:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:12:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:12:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:12:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:12:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:12:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:12:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:12:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:12:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:12:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:12:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:12:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:12:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:12:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:12:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:12:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:12:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:12:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:12:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:12:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:12:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:12:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:12:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:12:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:12:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:12:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:12:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:13:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:13:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:13:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:13:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:13:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:13:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:13:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:13:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:13:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:13:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:13:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:13:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:13:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:13:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:13:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:13:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:13:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:13:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:13:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:13:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:13:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:13:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:13:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:13:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:13:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:13:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:13:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:13:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:14:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:14:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:14:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:14:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:14:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:14:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:14:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:14:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:14:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:14:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:14:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:14:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:14:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:14:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:14:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:14:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:14:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:14:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:14:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:14:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:14:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:14:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:14:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:14:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:14:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:14:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:14:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:15:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:15:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:15:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:15:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:15:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:15:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:15:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:15:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:15:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:15:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:15:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:15:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:15:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:15:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:15:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:15:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:15:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:15:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:15:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:15:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:15:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:15:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:15:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:15:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:15:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:15:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:15:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:16:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:16:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:16:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:16:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:16:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:16:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:16:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:16:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:16:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:16:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:16:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:16:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:16:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:16:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:16:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:16:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:16:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:16:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:16:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:16:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:16:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:16:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:16:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:16:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:16:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:16:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:16:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:17:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:17:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:17:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:17:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:17:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:17:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:17:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:17:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:17:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:17:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:17:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:17:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:17:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:17:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:17:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:17:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:17:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:17:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:17:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:17:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:17:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:17:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:17:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:17:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:17:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:17:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:17:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:17:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:18:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:18:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:18:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:18:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:18:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:18:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:18:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:18:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:18:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:18:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:18:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:18:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:18:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:18:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:18:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:18:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:18:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:18:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:18:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:18:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:18:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:18:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:18:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:18:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:18:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:18:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:18:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:19:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:19:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:19:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:19:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:19:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:19:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:19:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:19:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:19:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:19:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:19:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:19:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:19:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:19:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:19:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:19:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:19:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:19:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:19:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:19:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:19:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:19:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:19:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:19:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:19:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:19:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:19:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:20:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:20:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:20:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:20:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:20:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:20:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:20:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:20:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:20:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:20:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:20:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:20:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:20:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:20:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:20:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:20:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:20:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:20:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:20:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:20:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:20:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:20:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:20:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:20:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:20:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:20:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:20:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:20:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:21:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:21:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:21:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:21:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:21:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:21:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:21:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:21:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:21:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:21:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:21:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:21:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:21:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:21:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:21:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:21:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:21:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:21:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:21:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:21:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:21:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:21:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:21:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:21:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:21:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:21:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:21:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:22:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:22:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:22:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:22:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:22:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:22:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:22:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:22:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:22:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:22:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:22:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:22:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:22:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:22:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:22:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:22:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:22:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:22:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:22:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:22:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:22:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:22:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:22:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:22:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:22:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:22:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:22:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:23:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:23:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:23:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:23:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:23:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:23:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:23:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:23:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:23:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:23:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:23:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:23:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:23:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:23:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:23:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:23:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:23:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:23:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:23:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:23:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:23:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:23:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:23:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:23:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:23:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:23:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:23:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:23:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:24:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:24:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:24:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:24:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:24:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:24:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:24:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:24:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:24:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:24:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:24:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:24:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:24:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:24:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:24:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:24:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:24:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:24:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:24:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:24:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:24:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:24:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:24:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:24:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:24:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:24:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:24:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:25:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:25:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:25:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:25:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:25:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:25:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:25:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:25:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:25:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:25:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:25:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:25:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:25:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:25:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:25:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:25:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:25:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:25:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:25:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:25:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:25:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:25:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:25:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:25:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:25:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:25:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:25:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:26:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:26:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:26:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:26:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:26:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:26:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:26:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:26:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:26:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:26:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:26:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:26:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:26:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:26:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:26:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:26:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:26:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:26:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:26:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:26:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:26:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:26:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:26:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:26:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:26:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:26:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:26:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:26:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:27:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:27:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:27:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:27:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:27:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:27:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:27:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:27:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:27:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:27:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:27:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:27:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:27:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:27:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:27:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:27:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:27:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:27:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:27:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:27:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:27:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:27:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:27:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:27:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:27:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:27:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:27:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:28:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:28:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:28:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:28:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:28:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:28:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:28:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:28:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:28:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:28:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:28:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:28:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:28:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:28:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:28:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:28:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:28:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:28:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:28:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:28:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:28:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:28:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:28:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:28:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:28:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:28:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:28:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:29:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:29:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:29:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:29:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:29:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:29:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:29:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:29:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:29:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:29:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:29:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:29:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:29:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:29:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:29:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:29:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:29:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:29:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:29:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:29:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:29:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:29:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:29:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:29:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:29:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:29:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:29:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:29:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:30:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:30:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:30:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:30:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:30:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:30:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:30:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:30:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:30:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:30:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:30:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:30:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:30:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:30:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:30:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:30:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:30:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:30:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:30:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:30:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:30:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:30:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:30:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:30:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:30:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:30:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:30:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:31:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:31:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:31:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:31:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:31:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:31:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:31:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:31:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:31:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:31:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:31:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:31:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:31:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:31:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:31:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:31:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:31:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:31:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:31:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:31:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:31:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:31:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:31:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:31:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:31:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:31:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:31:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:32:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:32:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:32:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:32:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:32:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:32:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:32:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:32:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:32:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:32:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:32:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:32:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:32:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:32:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:32:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:32:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:32:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:32:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:32:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:32:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:32:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:32:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:32:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:32:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:32:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:32:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:32:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:33:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:33:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:33:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:33:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:33:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:33:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:33:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:33:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:33:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:33:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:33:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:33:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:33:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:33:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:33:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:33:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:33:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:33:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:33:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:33:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:33:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:33:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:33:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:33:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:33:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:33:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:33:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:33:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:34:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:34:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:34:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:34:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:34:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:34:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:34:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:34:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:34:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:34:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:34:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:34:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:34:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:34:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:34:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:34:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:34:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:34:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:34:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:34:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:34:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:34:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:34:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:34:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:34:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:34:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:34:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:35:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:35:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:35:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:35:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:35:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:35:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:35:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:35:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:35:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:35:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:35:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:35:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:35:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:35:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:35:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:35:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:35:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:35:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:35:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:35:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:35:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:35:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:35:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:35:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:35:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:35:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:35:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:36:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:36:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:36:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:36:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:36:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:36:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:36:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:36:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:36:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:36:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:36:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:36:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:36:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:36:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:36:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:36:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:36:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:36:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:36:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:36:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:36:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:36:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:36:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:36:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:36:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:36:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:36:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:36:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:37:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:37:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:37:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:37:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:37:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:37:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:37:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:37:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:37:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:37:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:37:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:37:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:37:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:37:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:37:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:37:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:37:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:37:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:37:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:37:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:37:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:37:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:37:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:37:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:37:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:37:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:37:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:38:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:38:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:38:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:38:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:38:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:38:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:38:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:38:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:38:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:38:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:38:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:38:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:38:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:38:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:38:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:38:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:38:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:38:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:38:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:38:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:38:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:38:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:38:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:38:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:38:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:38:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:38:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:39:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:39:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:39:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:39:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:39:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:39:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:39:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:39:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:39:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:39:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:39:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:39:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:39:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:39:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:39:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:39:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:39:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:39:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:39:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:39:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:39:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:39:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:39:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:39:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:39:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:39:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:39:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:39:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:40:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:40:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:40:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:40:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:40:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:40:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:40:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:40:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:40:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:40:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:40:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:40:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:40:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:40:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:40:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:40:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:40:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:40:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:40:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:40:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:40:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:40:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:40:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:40:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:40:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:40:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:40:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:41:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:41:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:41:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:41:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:41:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:41:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:41:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:41:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:41:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:41:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:41:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:41:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:41:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:41:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:41:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:41:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:41:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:41:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:41:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:41:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:41:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:41:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:41:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:41:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:41:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:41:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:41:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:42:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:42:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:42:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:42:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:42:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:42:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:42:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:42:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:42:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:42:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:42:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:42:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:42:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:42:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:42:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:42:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:42:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:42:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:42:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:42:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:42:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:42:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:42:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:42:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:42:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:42:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:42:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:42:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:43:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:43:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:43:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:43:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:43:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:43:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:43:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:43:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:43:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:43:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:43:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:43:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:43:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:43:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:43:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:43:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:43:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:43:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:43:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:43:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:43:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:43:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:43:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:43:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:43:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:43:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:43:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:44:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:44:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:44:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:44:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:44:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:44:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:44:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:44:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:44:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:44:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:44:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:44:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:44:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:44:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:44:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:44:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:44:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:44:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:44:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:44:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:44:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:44:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:44:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:44:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:44:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:44:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:44:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:45:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:45:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:45:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:45:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:45:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:45:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:45:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:45:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:45:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:45:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:45:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:45:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:45:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:45:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:45:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:45:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:45:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:45:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:45:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:45:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:45:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:45:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:45:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:45:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:45:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:45:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:45:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:45:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:46:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:46:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:46:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:46:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:46:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:46:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:46:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:46:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:46:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:46:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:46:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:46:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:46:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:46:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:46:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:46:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:46:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:46:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:46:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:46:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:46:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:46:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:46:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:46:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:46:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:46:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:46:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:47:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:47:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:47:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:47:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:47:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:47:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:47:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:47:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:47:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:47:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:47:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:47:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:47:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:47:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:47:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:47:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:47:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:47:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:47:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:47:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:47:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:47:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:47:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:47:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:47:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:47:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:47:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:48:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:48:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:48:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:48:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:48:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:48:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:48:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:48:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:48:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:48:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:48:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:48:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:48:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:48:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:48:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:48:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:48:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:48:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:48:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:48:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:48:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:48:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:48:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:48:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:48:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:48:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:48:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:48:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:49:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:49:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:49:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:49:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:49:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:49:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:49:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:49:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:49:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:49:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:49:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:49:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:49:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:49:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:49:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:49:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:49:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:49:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:49:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:49:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:49:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:49:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:49:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:49:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:49:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:49:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:49:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:50:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:50:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:50:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:50:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:50:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:50:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:50:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:50:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:50:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:50:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:50:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:50:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:50:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:50:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:50:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:50:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:50:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:50:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:50:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:50:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:50:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:50:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:50:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:50:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:50:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:50:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:50:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:51:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:51:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:51:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:51:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:51:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:51:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:51:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:51:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:51:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:51:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:51:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:51:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:51:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:51:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:51:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:51:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:51:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:51:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:51:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:51:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:51:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:51:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:51:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:51:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:51:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:51:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:51:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:52:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:52:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:52:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:52:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:52:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:52:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:52:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:52:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:52:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:52:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:52:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:52:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:52:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:52:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:52:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:52:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:52:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:52:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:52:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:52:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:52:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:52:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:52:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:52:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:52:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:52:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:52:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:52:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:53:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:53:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:53:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:53:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:53:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:53:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:53:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:53:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:53:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:53:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:53:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:53:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:53:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:53:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:53:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:53:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:53:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:53:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:53:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:53:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:53:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:53:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:53:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:53:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:53:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:53:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:53:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:54:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:54:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:54:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:54:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:54:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:54:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:54:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:54:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:54:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:54:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:54:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:54:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:54:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:54:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:54:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:54:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:54:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:54:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:54:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:54:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:54:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:54:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:54:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:54:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:54:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:54:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:54:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:55:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:55:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:55:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:55:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:55:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:55:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:55:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:55:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:55:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:55:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:55:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:55:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:55:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:55:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:55:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:55:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:55:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:55:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:55:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:55:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:55:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:55:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:55:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:55:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:55:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:55:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:55:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:55:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:56:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:56:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:56:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:56:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:56:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:56:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:56:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:56:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:56:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:56:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:56:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:56:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:56:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:56:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:56:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:56:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:56:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:56:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:56:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:56:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:56:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:56:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:56:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:56:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:56:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:56:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:56:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:57:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:57:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:57:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:57:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:57:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:57:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:57:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:57:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:57:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:57:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:57:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:57:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:57:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:57:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:57:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:57:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:57:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:57:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:57:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:57:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:57:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:57:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:57:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:57:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:57:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:57:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:57:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:58:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:58:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:58:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:58:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:58:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:58:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:58:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:58:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:58:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:58:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:58:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:58:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:58:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:58:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:58:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:58:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:58:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:58:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:58:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:58:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:58:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:58:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:58:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:58:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:58:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:58:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:58:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:58:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:59:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:59:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:59:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:59:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:59:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:59:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:59:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:59:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:59:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:59:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:59:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:59:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:59:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:59:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:59:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:59:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:59:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:59:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:59:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:59:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:59:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:59:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 05:59:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 05:59:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 05:59:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 05:59:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 05:59:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:00:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:00:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:00:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:00:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:00:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:00:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:00:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:00:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:00:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:00:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:00:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:00:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:00:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:00:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:00:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:00:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:00:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:00:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:00:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:00:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:00:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:00:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:00:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:00:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:00:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:00:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:00:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:01:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:01:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:01:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:01:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:01:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:01:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:01:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:01:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:01:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:01:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:01:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:01:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:01:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:01:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:01:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:01:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:01:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:01:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:01:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:01:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:01:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:01:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:01:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:01:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:01:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:01:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:01:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:01:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:02:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:02:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:02:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:02:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:02:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:02:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:02:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:02:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:02:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:02:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:02:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:02:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:02:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:02:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:02:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:02:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:02:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:02:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:02:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:02:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:02:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:02:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:02:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:02:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:02:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:02:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:02:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:03:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:03:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:03:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:03:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:03:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:03:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:03:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:03:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:03:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:03:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:03:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:03:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:03:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:03:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:03:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:03:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:03:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:03:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:03:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:03:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:03:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:03:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:03:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:03:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:03:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:03:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:03:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:04:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:04:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:04:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:04:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:04:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:04:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:04:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:04:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:04:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:04:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:04:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:04:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:04:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:04:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:04:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:04:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:04:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:04:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:04:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:04:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:04:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:04:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:04:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:04:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:04:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:04:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:04:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:04:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:05:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:05:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:05:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:05:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:05:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:05:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:05:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:05:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:05:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:05:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:05:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:05:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:05:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:05:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:05:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:05:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:05:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:05:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:05:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:05:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:05:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:05:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:05:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:05:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:05:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:05:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:05:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:06:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:06:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:06:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:06:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:06:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:06:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:06:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:06:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:06:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:06:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:06:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:06:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:06:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:06:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:06:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:06:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:06:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:06:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:06:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:06:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:06:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:06:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:06:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:06:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:06:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:06:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:06:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:07:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:07:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:07:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:07:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:07:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:07:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:07:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:07:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:07:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:07:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:07:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:07:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:07:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:07:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:07:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:07:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:07:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:07:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:07:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:07:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:07:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:07:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:07:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:07:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:07:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:07:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:07:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:08:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:08:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:08:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:08:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:08:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:08:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:08:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:08:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:08:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:08:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:08:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:08:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:08:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:08:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:08:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:08:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:08:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:08:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:08:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:08:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:08:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:08:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:08:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:08:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:08:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:08:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:08:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:08:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:09:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:09:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:09:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:09:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:09:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:09:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:09:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:09:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:09:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:09:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:09:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:09:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:09:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:09:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:09:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:09:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:09:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:09:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:09:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:09:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:09:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:09:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:09:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:09:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:09:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:09:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:09:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:10:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:10:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:10:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:10:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:10:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:10:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:10:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:10:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:10:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:10:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:10:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:10:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:10:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:10:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:10:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:10:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:10:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:10:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:10:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:10:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:10:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:10:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:10:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:10:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:10:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:10:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:10:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:11:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:11:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:11:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:11:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:11:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:11:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:11:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:11:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:11:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:11:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:11:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:11:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:11:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:11:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:11:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:11:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:11:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:11:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:11:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:11:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:11:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:11:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:11:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:11:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:11:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:11:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:11:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:11:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:12:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:12:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:12:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:12:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:12:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:12:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:12:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:12:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:12:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:12:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:12:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:12:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:12:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:12:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:12:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:12:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:12:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:12:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:12:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:12:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:12:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:12:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:12:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:12:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:12:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:12:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:12:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:13:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:13:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:13:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:13:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:13:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:13:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:13:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:13:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:13:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:13:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:13:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:13:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:13:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:13:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:13:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:13:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:13:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:13:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:13:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:13:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:13:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:13:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:13:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:13:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:13:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:13:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:13:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:14:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:14:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:14:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:14:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:14:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:14:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:14:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:14:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:14:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:14:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:14:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:14:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:14:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:14:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:14:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:14:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:14:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:14:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:14:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:14:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:14:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:14:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:14:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:14:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:14:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:14:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:14:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:14:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:15:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:15:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:15:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:15:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:15:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:15:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:15:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:15:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:15:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:15:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:15:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:15:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:15:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:15:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:15:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:15:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:15:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:15:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:15:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:15:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:15:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:15:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:15:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:15:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:15:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:15:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:15:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:16:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:16:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:16:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:16:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:16:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:16:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:16:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:16:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:16:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:16:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:16:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:16:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:16:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:16:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:16:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:16:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:16:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:16:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:16:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:16:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:16:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:16:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:16:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:16:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:16:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:16:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:16:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:17:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:17:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:17:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:17:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:17:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:17:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:17:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:17:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:17:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:17:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:17:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:17:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:17:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:17:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:17:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:17:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:17:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:17:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:17:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:17:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:17:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:17:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:17:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:17:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:17:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:17:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:17:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:17:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:18:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:18:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:18:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:18:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:18:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:18:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:18:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:18:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:18:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:18:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:18:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:18:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:18:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:18:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:18:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:18:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:18:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:18:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:18:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:18:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:18:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:18:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:18:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:18:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:18:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:18:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:18:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:19:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:19:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:19:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:19:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:19:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:19:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:19:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:19:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:19:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:19:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:19:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:19:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:19:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:19:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:19:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:19:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:19:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:19:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:19:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:19:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:19:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:19:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:19:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:19:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:19:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:19:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:19:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:20:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:20:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:20:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:20:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:20:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:20:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:20:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:20:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:20:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:20:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:20:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:20:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:20:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:20:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:20:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:20:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:20:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:20:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:20:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:20:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:20:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:20:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:20:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:20:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:20:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:20:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:20:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:20:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:21:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:21:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:21:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:21:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:21:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:21:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:21:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:21:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:21:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:21:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:21:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:21:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:21:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:21:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:21:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:21:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:21:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:21:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:21:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:21:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:21:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:21:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:21:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:21:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:21:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:21:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:21:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:22:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:22:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:22:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:22:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:22:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:22:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:22:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:22:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:22:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:22:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:22:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:22:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:22:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:22:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:22:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:22:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:22:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:22:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:22:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:22:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:22:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:22:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:22:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:22:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:22:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:22:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:22:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:23:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:23:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:23:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:23:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:23:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:23:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:23:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:23:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:23:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:23:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:23:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:23:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:23:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:23:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:23:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:23:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:23:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:23:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:23:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:23:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:23:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:23:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:23:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:23:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:23:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:23:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:23:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:24:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:24:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:24:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:24:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:24:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:24:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:24:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:24:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:24:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:24:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:24:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:24:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:24:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:24:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:24:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:24:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:24:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:24:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:24:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:24:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:24:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:24:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:24:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:24:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:24:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:24:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:24:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:24:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:25:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:25:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:25:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:25:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:25:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:25:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:25:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:25:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:25:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:25:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:25:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:25:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:25:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:25:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:25:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:25:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:25:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:25:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:25:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:25:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:25:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:25:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:25:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:25:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:25:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:25:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:25:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:26:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  7 06:26:01 rdma06 dnf[563780]: Updating Subscription Management repositories.
Apr  7 06:26:01 rdma06 dnf[563780]: Unable to read consumer identity
Apr  7 06:26:01 rdma06 dnf[563780]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  7 06:26:01 rdma06 dnf[563780]: Metadata cache refreshed recently.
Apr  7 06:26:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  7 06:26:01 rdma06 systemd[1]: Finished dnf makecache.
Apr  7 06:26:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:26:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:26:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:26:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:26:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:26:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:26:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:26:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:26:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:26:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:26:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:26:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:26:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:26:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:26:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:26:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:26:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:26:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:26:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:26:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:26:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:26:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:26:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:26:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:26:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:26:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:26:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:27:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:27:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:27:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:27:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:27:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:27:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:27:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:27:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:27:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:27:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:27:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:27:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:27:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:27:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:27:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:27:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:27:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:27:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:27:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:27:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:27:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:27:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:27:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:27:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:27:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:27:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:27:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:27:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:28:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:28:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:28:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:28:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:28:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:28:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:28:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:28:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:28:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:28:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:28:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:28:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:28:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:28:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:28:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:28:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:28:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:28:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:28:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:28:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:28:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:28:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:28:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:28:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:28:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:28:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:28:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:29:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:29:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:29:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:29:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:29:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:29:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:29:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:29:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:29:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:29:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:29:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:29:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:29:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:29:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:29:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:29:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:29:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:29:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:29:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:29:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:29:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:29:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:29:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:29:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:29:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:29:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:29:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:30:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:30:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:30:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:30:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:30:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:30:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:30:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:30:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:30:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:30:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:30:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:30:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:30:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:30:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:30:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:30:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:30:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:30:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:30:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:30:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:30:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:30:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:30:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:30:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:30:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:30:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:30:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:30:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:31:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:31:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:31:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:31:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:31:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:31:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:31:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:31:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:31:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:31:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:31:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:31:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:31:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:31:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:31:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:31:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:31:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:31:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:31:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:31:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:31:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:31:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:31:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:31:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:31:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:31:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:31:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:32:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:32:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:32:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:32:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:32:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:32:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:32:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:32:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:32:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:32:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:32:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:32:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:32:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:32:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:32:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:32:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:32:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:32:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:32:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:32:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:32:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:32:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:32:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:32:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:32:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:32:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:32:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:33:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:33:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:33:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:33:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:33:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:33:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:33:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:33:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:33:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:33:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:33:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:33:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:33:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:33:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:33:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:33:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:33:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:33:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:33:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:33:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:33:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:33:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:33:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:33:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:33:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:33:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:33:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:33:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:34:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:34:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:34:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:34:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:34:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:34:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:34:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:34:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:34:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:34:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:34:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:34:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:34:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:34:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:34:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:34:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:34:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:34:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:34:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:34:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:34:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:34:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:34:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:34:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:34:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:34:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:34:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:35:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:35:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:35:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:35:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:35:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:35:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:35:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:35:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:35:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:35:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:35:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:35:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:35:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:35:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:35:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:35:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:35:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:35:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:35:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:35:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:35:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:35:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:35:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:35:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:35:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:35:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:35:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:36:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:36:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:36:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:36:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:36:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:36:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:36:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:36:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:36:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:36:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:36:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:36:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:36:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:36:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:36:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:36:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:36:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:36:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:36:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:36:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:36:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:36:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:36:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:36:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:36:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:36:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:36:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:36:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:37:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:37:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:37:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:37:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:37:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:37:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:37:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:37:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:37:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:37:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:37:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:37:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:37:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:37:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:37:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:37:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:37:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:37:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:37:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:37:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:37:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:37:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:37:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:37:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:37:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:37:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:37:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:38:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:38:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:38:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:38:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:38:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:38:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:38:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:38:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:38:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:38:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:38:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:38:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:38:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:38:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:38:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:38:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:38:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:38:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:38:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:38:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:38:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:38:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:38:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:38:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:38:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:38:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:38:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:39:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:39:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:39:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:39:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:39:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:39:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:39:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:39:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:39:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:39:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:39:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:39:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:39:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:39:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:39:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:39:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:39:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:39:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:39:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:39:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:39:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:39:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:39:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:39:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:39:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:39:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:39:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:40:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:40:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:40:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:40:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:40:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:40:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:40:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:40:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:40:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:40:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:40:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:40:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:40:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:40:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:40:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:40:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:40:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:40:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:40:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:40:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:40:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:40:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:40:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:40:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:40:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:40:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:40:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:40:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:41:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:41:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:41:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:41:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:41:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:41:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:41:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:41:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:41:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:41:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:41:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:41:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:41:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:41:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:41:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:41:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:41:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:41:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:41:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:41:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:41:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:41:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:41:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:41:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:41:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:41:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:41:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:42:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:42:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:42:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:42:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:42:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:42:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:42:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:42:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:42:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:42:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:42:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:42:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:42:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:42:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:42:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:42:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:42:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:42:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:42:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:42:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:42:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:42:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:42:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:42:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:42:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:42:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:42:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:43:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:43:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:43:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:43:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:43:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:43:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:43:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:43:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:43:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:43:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:43:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:43:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:43:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:43:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:43:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:43:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:43:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:43:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:43:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:43:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:43:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:43:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:43:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:43:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:43:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:43:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:43:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:43:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:44:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:44:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:44:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:44:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:44:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:44:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:44:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:44:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:44:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:44:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:44:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:44:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:44:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:44:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:44:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:44:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:44:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:44:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:44:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:44:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:44:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:44:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:44:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:44:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:44:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:44:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:44:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:45:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:45:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:45:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:45:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:45:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:45:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:45:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:45:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:45:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:45:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:45:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:45:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:45:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:45:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:45:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:45:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:45:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:45:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:45:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:45:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:45:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:45:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:45:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:45:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:45:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:45:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:45:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:46:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:46:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:46:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:46:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:46:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:46:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:46:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:46:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:46:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:46:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:46:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:46:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:46:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:46:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:46:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:46:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:46:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:46:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:46:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:46:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:46:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:46:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:46:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:46:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:46:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:46:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:46:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:46:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:47:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:47:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:47:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:47:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:47:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:47:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:47:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:47:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:47:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:47:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:47:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:47:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:47:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:47:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:47:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:47:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:47:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:47:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:47:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:47:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:47:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:47:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:47:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:47:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:47:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:47:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:47:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:48:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:48:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:48:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:48:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:48:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:48:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:48:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:48:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:48:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:48:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:48:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:48:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:48:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:48:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:48:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:48:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:48:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:48:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:48:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:48:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:48:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:48:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:48:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:48:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:48:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:48:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:48:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:49:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:49:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:49:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:49:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:49:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:49:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:49:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:49:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:49:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:49:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:49:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:49:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:49:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:49:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:49:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:49:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:49:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:49:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:49:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:49:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:49:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:49:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:49:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:49:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:49:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:49:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:49:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:49:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:50:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:50:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:50:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:50:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:50:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:50:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:50:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:50:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:50:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:50:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:50:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:50:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:50:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:50:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:50:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:50:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:50:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:50:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:50:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:50:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:50:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:50:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:50:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:50:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:50:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:50:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:50:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:51:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:51:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:51:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:51:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:51:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:51:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:51:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:51:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:51:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:51:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:51:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:51:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:51:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:51:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:51:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:51:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:51:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:51:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:51:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:51:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:51:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:51:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:51:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:51:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:51:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:51:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:51:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:52:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:52:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:52:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:52:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:52:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:52:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:52:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:52:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:52:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:52:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:52:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:52:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:52:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:52:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:52:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:52:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:52:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:52:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:52:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:52:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:52:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:52:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:52:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:52:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:52:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:52:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:52:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:52:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:53:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:53:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:53:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:53:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:53:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:53:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:53:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:53:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:53:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:53:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:53:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:53:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:53:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:53:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:53:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:53:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:53:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:53:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:53:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:53:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:53:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:53:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:53:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:53:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:53:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:53:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:53:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:54:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:54:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:54:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:54:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:54:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:54:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:54:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:54:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:54:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:54:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:54:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:54:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:54:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:54:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:54:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:54:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:54:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:54:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:54:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:54:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:54:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:54:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:54:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:54:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:54:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:54:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:54:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:55:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:55:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:55:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:55:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:55:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:55:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:55:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:55:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:55:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:55:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:55:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:55:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:55:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:55:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:55:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:55:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:55:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:55:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:55:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:55:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:55:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:55:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:55:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:55:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:55:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:55:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:55:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:55:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:56:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:56:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:56:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:56:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:56:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:56:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:56:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:56:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:56:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:56:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:56:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:56:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:56:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:56:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:56:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:56:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:56:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:56:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:56:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:56:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:56:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:56:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:56:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:56:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:56:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:56:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:56:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:57:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:57:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:57:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:57:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:57:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:57:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:57:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:57:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:57:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:57:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:57:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:57:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:57:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:57:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:57:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:57:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:57:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:57:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:57:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:57:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:57:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:57:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:57:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:57:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:57:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:57:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:57:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:58:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:58:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:58:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:58:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:58:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:58:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:58:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:58:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:58:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:58:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:58:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:58:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:58:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:58:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:58:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:58:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:58:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:58:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:58:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:58:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:58:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:58:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:58:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:58:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:58:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:58:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:58:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:59:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:59:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:59:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:59:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:59:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:59:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:59:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:59:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:59:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:59:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:59:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:59:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:59:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:59:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:59:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:59:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:59:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:59:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:59:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:59:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:59:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:59:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:59:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 06:59:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 06:59:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 06:59:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 06:59:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 06:59:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:00:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:00:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:00:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:00:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:00:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:00:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:00:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:00:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:00:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:00:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:00:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:00:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:00:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:00:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:00:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:00:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:00:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:00:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:00:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:00:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:00:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:00:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:00:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:00:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:00:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:00:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:00:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:01:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:01:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:01:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:01:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:01:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:01:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:01:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:01:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:01:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:01:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:01:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:01:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:01:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:01:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:01:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:01:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:01:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:01:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:01:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:01:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:01:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:01:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:01:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:01:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:01:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:01:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:01:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:02:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:02:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:02:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:02:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:02:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:02:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:02:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:02:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:02:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:02:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:02:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:02:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:02:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:02:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:02:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:02:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:02:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:02:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:02:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:02:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:02:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:02:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:02:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:02:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:02:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:02:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:02:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:02:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:03:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:03:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:03:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:03:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:03:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:03:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:03:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:03:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:03:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:03:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:03:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:03:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:03:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:03:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:03:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:03:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:03:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:03:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:03:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:03:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:03:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:03:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:03:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:03:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:03:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:03:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:03:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:04:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:04:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:04:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:04:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:04:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:04:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:04:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:04:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:04:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:04:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:04:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:04:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:04:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:04:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:04:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:04:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:04:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:04:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:04:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:04:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:04:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:04:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:04:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:04:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:04:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:04:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:04:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:05:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:05:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:05:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:05:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:05:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:05:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:05:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:05:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:05:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:05:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:05:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:05:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:05:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:05:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:05:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:05:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:05:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:05:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:05:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:05:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:05:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:05:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:05:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:05:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:05:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:05:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:05:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:05:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:06:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:06:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:06:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:06:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:06:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:06:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:06:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:06:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:06:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:06:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:06:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:06:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:06:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:06:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:06:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:06:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:06:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:06:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:06:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:06:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:06:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:06:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:06:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:06:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:06:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:06:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:06:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:07:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:07:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:07:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:07:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:07:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:07:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:07:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:07:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:07:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:07:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:07:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:07:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:07:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:07:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:07:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:07:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:07:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:07:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:07:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:07:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:07:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:07:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:07:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:07:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:07:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:07:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:07:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:08:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:08:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:08:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:08:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:08:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:08:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:08:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:08:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:08:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:08:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:08:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:08:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:08:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:08:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:08:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:08:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:08:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:08:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:08:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:08:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:08:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:08:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:08:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:08:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:08:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:08:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:08:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:08:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:09:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:09:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:09:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:09:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:09:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:09:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:09:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:09:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:09:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:09:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:09:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:09:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:09:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:09:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:09:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:09:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:09:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:09:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:09:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:09:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:09:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:09:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:09:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:09:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:09:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:09:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:09:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:10:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:10:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:10:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:10:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:10:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:10:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:10:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:10:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:10:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:10:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:10:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:10:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:10:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:10:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:10:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:10:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:10:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:10:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:10:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:10:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:10:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:10:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:10:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:10:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:10:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:10:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:10:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:11:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:11:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:11:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:11:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:11:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:11:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:11:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:11:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:11:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:11:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:11:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:11:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:11:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:11:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:11:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:11:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:11:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:11:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:11:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:11:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:11:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:11:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:11:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:11:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:11:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:11:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:11:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:11:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:12:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:12:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:12:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:12:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:12:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:12:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:12:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:12:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:12:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:12:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:12:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:12:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:12:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:12:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:12:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:12:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:12:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:12:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:12:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:12:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:12:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:12:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:12:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:12:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:12:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:12:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:12:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:13:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:13:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:13:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:13:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:13:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:13:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:13:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:13:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:13:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:13:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:13:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:13:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:13:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:13:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:13:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:13:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:13:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:13:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:13:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:13:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:13:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:13:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:13:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:13:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:13:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:13:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:13:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:14:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:14:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:14:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:14:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:14:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:14:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:14:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:14:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:14:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:14:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:14:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:14:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:14:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:14:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:14:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:14:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:14:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:14:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:14:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:14:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:14:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:14:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:14:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:14:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:14:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:14:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:14:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:15:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:15:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:15:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:15:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:15:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:15:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:15:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:15:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:15:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:15:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:15:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:15:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:15:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:15:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:15:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:15:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:15:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:15:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:15:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:15:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:15:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:15:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:15:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:15:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:15:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:15:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:15:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:15:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:16:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:16:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:16:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:16:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:16:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:16:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:16:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:16:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:16:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:16:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:16:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:16:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:16:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:16:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:16:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:16:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:16:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:16:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:16:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:16:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:16:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:16:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:16:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:16:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:16:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:16:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:16:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:17:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:17:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:17:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:17:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:17:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:17:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:17:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:17:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:17:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:17:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:17:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:17:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:17:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:17:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:17:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:17:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:17:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:17:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:17:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:17:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:17:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:17:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:17:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:17:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:17:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:17:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:17:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:18:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:18:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:18:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:18:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:18:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:18:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:18:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:18:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:18:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:18:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:18:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:18:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:18:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:18:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:18:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:18:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:18:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:18:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:18:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:18:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:18:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:18:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:18:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:18:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:18:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:18:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:18:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:18:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:19:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:19:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:19:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:19:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:19:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:19:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:19:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:19:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:19:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:19:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:19:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:19:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:19:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:19:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:19:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:19:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:19:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:19:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:19:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:19:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:19:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:19:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:19:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:19:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:19:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:19:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:19:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:20:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:20:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:20:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:20:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:20:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:20:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:20:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:20:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:20:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:20:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:20:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:20:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:20:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:20:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:20:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:20:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:20:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:20:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:20:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:20:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:20:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:20:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:20:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:20:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:20:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:20:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:20:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:21:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:21:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:21:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:21:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:21:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:21:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:21:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:21:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:21:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:21:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:21:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:21:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:21:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:21:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:21:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:21:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:21:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:21:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:21:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:21:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:21:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:21:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:21:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:21:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:21:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:21:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:21:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:21:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:22:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:22:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:22:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:22:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:22:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:22:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:22:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:22:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:22:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:22:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:22:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:22:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:22:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:22:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:22:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:22:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:22:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:22:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:22:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:22:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:22:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:22:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:22:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:22:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:22:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:22:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:22:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:23:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:23:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:23:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:23:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:23:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:23:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:23:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:23:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:23:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:23:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:23:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:23:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:23:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:23:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:23:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:23:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:23:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:23:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:23:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:23:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:23:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:23:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:23:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:23:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:23:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:23:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:23:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:24:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:24:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:24:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:24:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:24:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:24:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:24:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:24:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:24:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:24:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:24:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:24:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:24:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:24:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:24:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:24:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:24:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:24:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:24:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:24:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:24:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:24:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:24:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:24:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:24:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:24:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:24:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:24:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:25:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:25:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:25:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:25:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:25:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:25:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:25:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:25:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:25:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:25:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:25:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:25:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:25:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:25:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:25:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:25:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:25:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:25:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:25:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:25:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:25:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:25:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:25:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:25:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:25:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:25:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:25:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:26:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:26:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:26:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:26:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:26:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:26:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:26:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:26:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:26:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:26:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:26:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:26:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:26:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:26:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:26:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:26:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:26:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:26:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:26:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:26:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:26:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:26:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:26:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:26:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:26:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:26:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:26:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:27:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:27:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:27:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:27:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:27:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:27:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:27:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:27:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:27:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:27:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:27:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:27:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:27:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:27:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:27:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:27:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:27:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:27:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:27:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:27:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:27:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:27:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:27:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:27:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:27:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:27:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:27:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:27:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:28:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:28:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:28:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:28:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:28:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:28:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:28:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:28:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:28:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:28:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:28:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:28:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:28:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:28:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:28:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:28:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:28:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:28:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:28:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:28:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:28:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:28:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:28:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:28:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:28:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:28:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:28:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:29:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:29:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:29:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:29:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:29:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:29:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:29:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:29:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:29:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:29:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:29:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:29:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:29:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:29:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:29:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:29:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:29:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:29:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:29:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:29:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:29:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:29:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:29:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:29:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:29:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:29:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:29:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:30:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:30:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:30:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:30:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:30:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:30:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:30:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:30:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:30:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:30:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:30:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:30:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:30:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:30:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:30:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:30:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:30:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:30:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:30:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:30:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:30:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:30:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:30:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:30:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:30:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:30:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:30:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:31:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:31:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:31:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:31:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:31:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:31:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:31:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:31:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:31:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:31:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:31:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:31:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:31:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:31:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:31:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:31:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:31:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:31:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:31:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:31:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:31:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:31:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:31:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:31:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:31:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:31:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:31:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:31:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:32:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:32:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:32:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:32:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:32:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:32:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:32:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:32:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:32:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:32:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:32:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:32:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:32:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:32:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:32:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:32:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:32:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:32:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:32:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:32:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:32:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:32:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:32:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:32:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:32:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:32:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:32:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:33:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:33:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:33:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:33:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:33:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:33:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:33:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:33:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:33:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:33:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:33:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:33:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:33:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:33:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:33:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:33:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:33:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:33:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:33:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:33:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:33:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:33:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:33:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:33:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:33:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:33:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:33:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:34:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:34:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:34:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:34:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:34:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:34:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:34:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:34:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:34:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:34:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:34:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:34:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:34:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:34:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:34:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:34:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:34:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:34:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:34:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:34:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:34:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:34:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:34:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:34:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:34:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:34:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:34:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:34:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:35:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:35:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:35:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:35:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:35:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:35:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:35:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:35:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:35:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:35:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:35:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:35:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:35:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:35:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:35:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:35:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:35:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:35:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:35:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:35:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:35:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:35:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:35:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:35:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:35:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:35:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:35:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:36:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:36:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:36:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:36:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:36:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:36:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:36:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:36:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:36:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:36:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:36:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:36:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:36:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:36:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:36:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:36:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:36:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:36:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:36:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:36:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:36:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:36:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:36:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:36:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:36:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:36:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:36:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:37:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:37:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:37:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:37:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:37:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:37:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:37:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:37:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:37:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:37:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:37:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:37:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:37:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:37:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:37:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:37:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:37:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:37:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:37:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:37:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:37:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:37:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:37:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:37:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:37:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:37:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:37:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:37:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:38:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:38:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:38:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:38:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:38:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:38:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:38:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:38:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:38:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:38:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:38:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:38:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:38:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:38:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:38:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:38:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:38:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:38:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:38:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:38:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:38:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:38:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:38:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:38:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:38:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:38:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:38:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:39:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:39:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:39:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:39:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:39:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:39:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:39:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:39:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:39:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:39:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:39:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:39:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:39:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:39:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:39:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:39:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:39:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:39:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:39:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:39:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:39:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:39:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:39:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:39:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:39:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:39:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:39:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:40:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:40:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:40:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:40:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:40:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:40:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:40:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:40:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:40:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:40:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:40:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:40:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:40:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:40:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:40:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:40:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:40:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:40:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:40:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:40:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:40:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:40:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:40:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:40:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:40:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:40:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:40:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:40:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:41:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:41:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:41:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:41:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:41:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:41:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:41:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:41:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:41:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:41:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:41:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:41:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:41:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:41:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:41:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:41:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:41:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:41:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:41:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:41:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:41:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:41:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:41:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:41:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:41:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:41:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:41:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:42:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:42:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:42:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:42:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:42:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:42:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:42:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:42:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:42:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:42:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:42:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:42:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:42:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:42:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:42:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:42:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:42:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:42:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:42:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:42:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:42:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:42:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:42:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:42:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:42:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:42:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:42:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:43:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:43:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:43:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:43:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:43:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:43:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:43:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:43:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:43:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:43:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:43:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:43:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:43:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:43:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:43:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:43:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:43:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:43:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:43:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:43:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:43:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:43:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:43:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:43:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:43:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:43:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:43:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:43:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:44:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:44:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:44:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:44:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:44:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:44:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:44:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:44:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:44:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:44:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:44:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:44:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:44:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:44:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:44:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:44:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:44:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:44:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:44:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:44:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:44:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:44:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:44:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:44:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:44:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:44:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:44:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:45:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:45:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:45:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:45:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:45:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:45:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:45:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:45:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:45:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:45:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:45:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:45:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:45:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:45:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:45:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:45:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:45:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:45:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:45:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:45:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:45:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:45:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:45:45 rdma06 systemd[1]: Starting dnf makecache...
Apr  7 07:45:46 rdma06 dnf[567009]: Updating Subscription Management repositories.
Apr  7 07:45:46 rdma06 dnf[567009]: Unable to read consumer identity
Apr  7 07:45:46 rdma06 dnf[567009]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  7 07:45:46 rdma06 dnf[567009]: Metadata cache refreshed recently.
Apr  7 07:45:46 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  7 07:45:46 rdma06 systemd[1]: Finished dnf makecache.
Apr  7 07:45:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:45:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:45:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:45:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:45:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:46:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:46:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:46:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:46:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:46:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:46:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:46:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:46:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:46:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:46:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:46:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:46:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:46:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:46:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:46:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:46:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:46:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:46:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:46:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:46:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:46:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:46:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:46:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:46:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:46:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:46:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:46:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:47:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:47:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:47:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:47:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:47:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:47:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:47:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:47:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:47:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:47:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:47:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:47:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:47:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:47:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:47:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:47:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:47:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:47:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:47:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:47:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:47:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:47:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:47:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:47:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:47:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:47:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:47:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:47:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:48:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:48:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:48:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:48:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:48:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:48:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:48:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:48:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:48:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:48:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:48:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:48:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:48:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:48:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:48:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:48:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:48:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:48:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:48:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:48:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:48:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:48:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:48:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:48:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:48:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:48:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:48:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:49:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:49:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:49:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:49:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:49:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:49:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:49:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:49:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:49:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:49:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:49:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:49:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:49:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:49:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:49:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:49:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:49:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:49:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:49:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:49:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:49:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:49:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:49:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:49:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:49:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:49:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:49:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:50:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:50:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:50:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:50:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:50:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:50:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:50:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:50:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:50:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:50:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:50:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:50:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:50:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:50:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:50:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:50:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:50:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:50:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:50:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:50:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:50:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:50:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:50:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:50:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:50:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:50:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:50:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:50:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:51:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:51:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:51:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:51:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:51:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:51:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:51:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:51:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:51:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:51:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:51:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:51:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:51:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:51:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:51:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:51:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:51:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:51:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:51:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:51:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:51:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:51:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:51:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:51:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:51:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:51:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:51:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:52:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:52:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:52:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:52:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:52:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:52:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:52:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:52:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:52:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:52:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:52:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:52:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:52:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:52:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:52:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:52:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:52:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:52:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:52:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:52:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:52:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:52:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:52:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:52:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:52:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:52:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:52:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:53:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:53:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:53:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:53:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:53:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:53:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:53:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:53:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:53:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:53:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:53:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:53:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:53:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:53:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:53:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:53:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:53:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:53:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:53:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:53:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:53:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:53:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:53:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:53:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:53:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:53:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:53:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:53:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:54:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:54:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:54:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:54:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:54:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:54:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:54:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:54:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:54:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:54:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:54:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:54:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:54:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:54:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:54:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:54:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:54:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:54:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:54:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:54:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:54:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:54:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:54:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:54:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:54:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:54:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:54:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:55:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:55:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:55:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:55:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:55:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:55:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:55:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:55:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:55:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:55:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:55:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:55:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:55:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:55:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:55:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:55:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:55:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:55:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:55:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:55:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:55:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:55:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:55:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:55:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:55:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:55:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:55:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:56:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:56:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:56:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:56:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:56:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:56:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:56:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:56:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:56:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:56:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:56:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:56:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:56:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:56:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:56:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:56:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:56:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:56:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:56:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:56:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:56:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:56:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:56:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:56:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:56:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:56:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:56:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:56:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:57:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:57:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:57:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:57:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:57:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:57:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:57:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:57:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:57:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:57:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:57:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:57:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:57:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:57:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:57:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:57:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:57:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:57:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:57:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:57:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:57:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:57:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:57:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:57:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:57:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:57:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:57:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:58:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:58:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:58:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:58:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:58:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:58:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:58:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:58:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:58:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:58:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:58:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:58:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:58:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:58:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:58:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:58:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:58:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:58:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:58:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:58:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:58:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:58:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:58:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:58:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:58:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:58:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:58:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:59:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:59:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:59:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:59:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:59:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:59:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:59:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:59:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:59:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:59:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:59:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:59:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:59:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:59:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:59:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:59:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:59:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:59:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:59:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 07:59:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:59:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:59:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:59:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:59:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 07:59:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 07:59:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 07:59:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 07:59:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:00:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:00:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:00:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:00:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:00:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:00:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:00:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:00:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:00:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:00:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:00:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:00:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:00:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:00:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:00:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:00:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:00:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:00:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:00:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:00:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:00:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:00:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:00:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:00:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:00:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:00:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:00:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:01:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:01:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:01:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:01:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:01:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:01:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:01:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:01:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:01:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:01:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:01:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:01:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:01:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:01:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:01:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:01:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:01:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:01:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:01:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:01:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:01:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:01:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:01:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:01:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:01:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:01:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:01:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:02:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:02:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:02:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:02:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:02:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:02:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:02:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:02:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:02:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:02:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:02:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:02:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:02:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:02:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:02:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:02:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:02:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:02:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:02:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:02:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:02:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:02:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:02:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:02:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:02:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:02:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:02:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:02:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:03:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:03:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:03:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:03:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:03:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:03:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:03:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:03:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:03:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:03:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:03:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:03:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:03:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:03:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:03:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:03:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:03:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:03:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:03:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:03:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:03:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:03:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:03:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:03:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:03:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:03:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:03:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:04:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:04:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:04:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:04:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:04:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:04:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:04:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:04:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:04:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:04:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:04:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:04:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:04:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:04:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:04:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:04:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:04:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:04:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:04:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:04:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:04:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:04:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:04:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:04:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:04:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:04:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:04:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:05:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:05:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:05:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:05:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:05:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:05:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:05:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:05:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:05:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:05:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:05:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:05:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:05:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:05:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:05:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:05:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:05:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:05:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:05:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:05:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:05:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:05:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:05:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:05:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:05:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:05:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:05:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:06:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:06:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:06:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:06:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:06:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:06:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:06:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:06:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:06:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:06:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:06:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:06:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:06:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:06:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:06:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:06:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:06:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:06:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:06:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:06:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:06:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:06:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:06:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:06:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:06:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:06:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:06:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:06:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:07:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:07:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:07:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:07:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:07:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:07:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:07:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:07:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:07:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:07:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:07:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:07:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:07:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:07:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:07:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:07:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:07:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:07:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:07:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:07:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:07:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:07:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:07:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:07:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:07:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:07:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:07:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:08:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:08:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:08:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:08:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:08:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:08:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:08:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:08:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:08:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:08:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:08:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:08:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:08:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:08:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:08:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:08:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:08:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:08:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:08:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:08:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:08:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:08:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:08:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:08:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:08:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:08:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:08:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:09:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:09:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:09:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:09:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:09:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:09:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:09:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:09:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:09:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:09:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:09:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:09:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:09:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:09:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:09:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:09:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:09:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:09:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:09:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:09:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:09:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:09:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:09:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:09:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:09:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:09:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:09:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:09:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:10:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:10:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:10:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:10:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:10:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:10:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:10:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:10:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:10:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:10:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:10:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:10:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:10:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:10:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:10:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:10:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:10:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:10:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:10:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:10:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:10:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:10:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:10:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:10:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:10:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:10:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:10:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:11:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:11:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:11:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:11:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:11:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:11:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:11:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:11:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:11:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:11:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:11:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:11:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:11:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:11:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:11:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:11:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:11:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:11:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:11:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:11:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:11:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:11:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:11:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:11:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:11:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:11:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:11:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:12:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:12:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:12:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:12:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:12:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:12:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:12:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:12:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:12:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:12:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:12:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:12:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:12:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:12:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:12:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:12:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:12:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:12:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:12:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:12:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:12:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:12:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:12:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:12:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:12:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:12:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:12:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:12:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:13:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:13:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:13:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:13:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:13:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:13:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:13:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:13:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:13:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:13:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:13:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:13:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:13:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:13:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:13:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:13:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:13:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:13:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:13:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:13:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:13:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:13:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:13:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:13:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:13:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:13:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:13:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:14:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:14:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:14:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:14:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:14:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:14:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:14:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:14:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:14:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:14:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:14:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:14:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:14:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:14:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:14:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:14:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:14:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:14:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:14:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:14:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:14:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:14:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:14:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:14:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:14:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:14:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:14:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:15:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:15:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:15:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:15:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:15:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:15:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:15:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:15:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:15:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:15:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:15:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:15:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:15:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:15:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:15:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:15:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:15:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:15:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:15:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:15:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:15:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:15:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:15:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:15:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:15:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:15:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:15:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:15:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:16:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:16:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:16:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:16:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:16:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:16:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:16:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:16:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:16:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:16:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:16:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:16:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:16:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:16:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:16:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:16:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:16:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:16:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:16:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:16:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:16:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:16:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:16:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:16:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:16:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:16:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:16:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:17:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:17:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:17:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:17:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:17:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:17:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:17:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:17:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:17:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:17:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:17:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:17:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:17:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:17:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:17:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:17:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:17:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:17:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:17:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:17:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:17:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:17:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:17:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:17:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:17:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:17:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:17:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:18:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:18:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:18:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:18:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:18:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:18:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:18:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:18:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:18:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:18:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:18:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:18:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:18:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:18:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:18:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:18:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:18:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:18:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:18:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:18:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:18:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:18:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:18:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:18:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:18:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:18:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:18:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:18:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:19:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:19:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:19:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:19:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:19:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:19:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:19:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:19:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:19:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:19:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:19:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:19:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:19:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:19:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:19:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:19:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:19:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:19:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:19:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:19:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:19:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:19:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:19:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:19:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:19:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:19:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:19:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:20:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:20:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:20:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:20:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:20:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:20:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:20:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:20:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:20:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:20:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:20:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:20:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:20:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:20:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:20:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:20:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:20:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:20:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:20:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:20:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:20:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:20:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:20:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:20:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:20:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:20:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:20:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:21:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:21:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:21:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:21:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:21:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:21:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:21:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:21:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:21:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:21:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:21:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:21:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:21:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:21:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:21:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:21:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:21:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:21:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:21:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:21:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:21:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:21:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:21:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:21:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:21:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:21:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:21:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:22:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:22:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:22:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:22:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:22:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:22:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:22:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:22:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:22:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:22:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:22:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:22:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:22:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:22:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:22:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:22:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:22:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:22:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:22:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:22:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:22:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:22:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:22:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:22:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:22:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:22:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:22:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:22:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:23:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:23:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:23:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:23:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:23:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:23:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:23:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:23:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:23:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:23:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:23:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:23:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:23:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:23:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:23:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:23:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:23:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:23:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:23:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:23:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:23:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:23:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:23:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:23:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:23:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:23:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:23:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:24:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:24:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:24:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:24:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:24:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:24:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:24:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:24:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:24:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:24:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:24:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:24:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:24:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:24:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:24:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:24:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:24:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:24:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:24:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:24:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:24:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:24:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:24:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:24:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:24:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:24:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:24:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:25:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:25:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:25:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:25:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:25:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:25:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:25:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:25:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:25:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:25:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:25:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:25:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:25:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:25:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:25:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:25:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:25:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:25:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:25:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:25:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:25:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:25:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:25:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:25:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:25:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:25:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:25:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:25:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:26:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:26:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:26:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:26:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:26:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:26:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:26:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:26:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:26:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:26:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:26:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:26:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:26:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:26:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:26:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:26:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:26:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:26:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:26:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:26:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:26:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:26:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:26:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:26:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:26:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:26:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:26:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:27:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:27:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:27:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:27:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:27:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:27:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:27:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:27:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:27:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:27:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:27:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:27:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:27:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:27:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:27:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:27:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:27:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:27:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:27:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:27:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:27:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:27:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:27:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:27:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:27:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:27:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:27:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:28:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:28:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:28:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:28:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:28:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:28:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:28:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:28:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:28:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:28:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:28:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:28:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:28:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:28:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:28:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:28:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:28:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:28:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:28:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:28:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:28:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:28:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:28:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:28:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:28:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:28:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:28:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:28:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:29:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:29:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:29:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:29:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:29:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:29:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:29:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:29:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:29:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:29:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:29:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:29:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:29:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:29:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:29:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:29:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:29:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:29:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:29:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:29:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:29:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:29:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:29:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:29:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:29:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:29:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:29:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:30:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:30:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:30:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:30:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:30:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:30:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:30:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:30:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:30:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:30:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:30:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:30:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:30:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:30:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:30:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:30:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:30:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:30:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:30:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:30:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:30:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:30:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:30:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:30:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:30:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:30:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:30:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:31:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:31:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:31:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:31:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:31:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:31:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:31:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:31:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:31:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:31:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:31:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:31:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:31:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:31:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:31:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:31:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:31:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:31:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:31:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:31:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:31:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:31:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:31:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:31:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:31:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:31:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:31:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:31:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:32:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:32:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:32:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:32:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:32:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:32:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:32:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:32:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:32:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:32:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:32:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:32:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:32:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:32:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:32:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:32:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:32:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:32:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:32:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:32:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:32:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:32:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:32:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:32:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:32:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:32:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:32:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:33:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:33:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:33:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:33:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:33:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:33:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:33:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:33:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:33:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:33:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:33:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:33:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:33:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:33:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:33:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:33:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:33:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:33:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:33:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:33:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:33:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:33:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:33:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:33:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:33:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:33:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:33:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:34:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:34:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:34:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:34:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:34:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:34:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:34:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:34:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:34:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:34:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:34:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:34:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:34:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:34:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:34:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:34:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:34:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:34:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:34:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:34:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:34:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:34:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:34:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:34:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:34:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:34:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:34:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:34:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:35:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:35:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:35:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:35:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:35:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:35:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:35:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:35:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:35:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:35:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:35:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:35:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:35:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:35:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:35:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:35:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:35:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:35:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:35:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:35:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:35:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:35:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:35:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:35:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:35:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:35:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:35:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:36:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:36:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:36:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:36:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:36:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:36:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:36:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:36:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:36:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:36:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:36:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:36:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:36:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:36:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:36:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:36:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:36:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:36:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:36:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:36:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:36:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:36:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:36:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:36:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:36:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:36:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:36:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:37:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:37:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:37:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:37:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:37:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:37:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:37:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:37:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:37:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:37:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:37:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:37:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:37:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:37:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:37:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:37:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:37:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:37:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:37:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:37:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:37:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:37:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:37:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:37:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:37:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:37:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:37:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:38:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:38:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:38:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:38:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:38:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:38:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:38:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:38:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:38:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:38:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:38:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:38:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:38:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:38:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:38:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:38:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:38:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:38:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:38:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:38:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:38:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:38:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:38:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:38:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:38:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:38:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:38:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:38:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:39:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:39:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:39:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:39:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:39:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:39:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:39:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:39:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:39:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:39:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:39:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:39:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:39:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:39:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:39:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:39:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:39:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:39:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:39:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:39:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:39:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:39:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:39:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:39:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:39:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:39:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:39:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:40:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:40:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:40:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:40:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:40:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:40:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:40:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:40:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:40:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:40:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:40:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:40:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:40:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:40:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:40:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:40:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:40:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:40:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:40:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:40:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:40:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:40:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:40:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:40:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:40:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:40:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:40:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:41:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:41:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:41:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:41:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:41:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:41:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:41:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:41:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:41:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:41:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:41:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:41:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:41:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:41:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:41:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:41:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:41:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:41:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:41:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:41:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:41:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:41:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:41:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:41:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:41:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:41:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:41:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:41:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:42:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:42:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:42:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:42:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:42:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:42:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:42:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:42:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:42:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:42:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:42:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:42:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:42:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:42:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:42:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:42:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:42:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:42:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:42:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:42:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:42:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:42:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:42:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:42:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:42:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:42:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:42:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:43:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:43:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:43:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:43:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:43:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:43:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:43:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:43:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:43:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:43:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:43:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:43:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:43:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:43:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:43:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:43:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:43:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:43:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:43:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:43:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:43:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:43:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:43:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:43:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:43:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:43:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:43:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:44:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:44:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:44:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:44:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:44:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:44:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:44:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:44:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:44:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:44:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:44:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:44:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:44:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:44:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:44:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:44:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:44:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:44:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:44:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:44:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:44:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:44:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:44:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:44:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:44:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:44:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:44:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:44:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:45:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:45:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:45:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:45:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:45:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:45:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:45:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:45:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:45:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:45:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:45:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:45:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:45:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:45:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:45:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:45:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:45:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:45:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:45:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:45:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:45:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:45:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:45:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:45:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:45:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:45:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:45:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:46:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:46:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:46:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:46:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:46:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:46:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:46:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:46:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:46:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:46:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:46:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:46:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:46:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:46:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:46:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:46:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:46:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:46:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:46:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:46:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:46:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:46:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:46:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:46:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:46:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:46:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:46:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:47:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:47:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:47:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:47:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:47:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:47:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:47:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:47:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:47:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:47:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:47:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:47:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:47:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:47:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:47:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:47:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:47:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:47:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:47:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:47:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:47:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:47:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:47:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:47:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:47:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:47:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:47:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:47:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:48:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:48:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:48:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:48:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:48:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:48:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:48:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:48:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:48:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:48:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:48:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:48:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:48:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:48:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:48:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:48:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:48:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:48:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:48:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:48:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:48:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:48:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:48:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:48:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:48:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:48:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:48:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:49:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:49:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:49:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:49:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:49:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:49:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:49:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:49:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:49:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:49:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:49:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:49:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:49:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:49:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:49:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:49:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:49:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:49:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:49:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:49:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:49:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:49:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:49:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:49:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:49:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:49:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:49:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:50:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:50:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:50:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:50:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:50:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:50:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:50:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:50:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:50:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:50:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:50:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:50:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:50:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:50:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:50:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:50:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:50:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:50:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:50:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:50:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:50:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:50:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:50:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:50:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:50:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:50:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:50:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:50:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:51:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:51:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:51:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:51:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:51:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:51:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:51:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:51:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:51:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:51:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:51:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:51:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:51:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:51:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:51:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:51:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:51:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:51:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:51:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:51:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:51:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:51:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:51:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:51:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:51:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:51:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:51:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:52:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:52:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:52:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:52:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:52:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:52:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:52:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:52:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:52:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:52:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:52:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:52:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:52:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:52:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:52:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:52:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:52:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:52:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:52:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:52:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:52:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:52:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:52:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:52:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:52:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:52:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:52:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:53:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:53:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:53:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:53:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:53:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:53:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:53:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:53:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:53:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:53:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:53:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:53:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:53:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:53:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:53:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:53:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:53:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:53:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:53:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:53:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:53:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:53:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:53:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:53:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:53:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:53:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:53:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:54:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:54:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:54:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:54:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:54:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:54:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:54:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:54:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:54:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:54:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:54:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:54:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:54:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:54:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:54:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:54:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:54:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:54:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:54:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:54:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:54:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:54:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:54:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:54:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:54:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:54:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:54:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:54:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:55:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:55:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:55:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:55:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:55:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:55:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:55:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:55:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:55:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:55:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:55:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:55:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:55:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:55:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:55:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:55:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:55:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:55:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:55:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:55:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:55:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:55:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:55:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:55:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:55:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:55:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:55:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:56:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:56:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:56:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:56:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:56:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:56:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:56:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:56:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:56:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:56:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:56:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:56:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:56:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:56:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:56:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:56:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:56:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:56:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:56:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:56:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:56:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:56:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:56:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:56:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:56:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:56:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:56:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:57:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:57:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:57:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:57:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:57:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:57:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:57:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:57:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:57:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:57:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:57:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:57:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:57:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:57:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:57:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:57:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:57:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:57:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:57:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:57:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:57:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:57:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:57:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:57:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:57:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:57:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:57:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:57:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:58:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:58:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:58:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:58:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:58:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:58:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:58:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:58:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:58:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:58:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:58:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:58:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:58:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:58:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:58:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:58:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:58:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:58:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:58:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:58:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:58:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:58:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:58:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:58:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:58:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:58:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:58:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:59:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:59:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:59:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:59:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:59:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:59:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:59:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:59:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:59:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:59:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:59:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:59:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:59:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:59:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:59:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:59:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:59:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:59:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:59:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 08:59:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:59:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:59:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:59:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 08:59:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 08:59:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 08:59:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 08:59:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:00:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:00:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:00:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:00:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:00:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:00:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:00:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:00:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:00:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:00:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:00:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:00:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:00:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:00:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:00:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:00:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:00:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:00:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:00:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:00:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:00:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:00:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:00:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:00:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:00:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:00:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:00:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:00:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:01:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:01:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:01:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:01:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:01:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:01:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:01:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:01:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:01:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:01:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:01:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:01:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:01:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:01:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:01:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:01:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:01:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:01:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:01:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:01:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:01:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:01:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:01:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:01:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:01:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:01:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:01:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:02:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:02:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:02:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:02:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:02:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:02:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:02:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:02:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:02:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:02:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:02:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:02:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:02:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:02:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:02:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:02:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:02:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:02:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:02:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:02:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:02:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:02:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:02:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:02:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:02:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:02:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:02:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:03:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:03:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:03:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:03:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:03:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:03:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:03:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:03:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:03:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:03:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:03:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:03:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:03:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:03:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:03:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:03:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:03:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:03:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:03:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:03:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:03:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:03:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:03:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:03:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:03:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:03:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:03:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:03:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:04:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:04:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:04:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:04:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:04:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:04:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:04:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:04:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:04:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:04:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:04:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:04:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:04:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:04:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:04:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:04:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:04:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:04:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:04:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:04:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:04:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:04:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:04:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:04:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:04:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:04:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:04:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:05:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:05:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:05:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:05:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:05:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:05:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:05:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:05:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:05:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:05:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:05:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:05:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:05:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:05:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:05:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:05:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:05:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:05:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:05:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:05:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:05:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:05:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:05:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:05:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:05:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:05:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:05:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:06:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:06:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:06:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:06:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:06:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:06:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:06:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:06:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:06:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:06:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:06:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:06:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:06:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:06:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:06:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:06:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:06:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:06:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:06:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:06:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:06:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:06:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:06:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:06:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:06:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:06:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:06:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:06:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:07:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:07:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:07:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:07:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:07:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:07:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:07:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:07:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:07:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:07:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:07:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:07:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:07:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:07:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:07:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:07:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:07:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:07:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:07:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:07:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:07:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:07:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:07:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:07:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:07:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:07:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:07:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:08:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:08:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:08:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:08:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:08:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:08:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:08:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:08:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:08:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:08:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:08:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:08:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:08:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:08:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:08:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:08:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:08:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:08:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:08:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:08:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:08:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:08:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:08:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:08:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:08:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:08:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:08:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:09:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:09:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:09:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:09:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:09:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:09:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:09:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:09:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:09:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:09:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:09:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:09:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:09:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:09:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:09:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:09:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:09:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:09:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:09:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:09:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:09:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:09:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:09:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:09:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:09:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:09:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:09:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:09:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:10:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:10:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:10:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:10:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:10:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:10:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:10:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:10:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:10:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:10:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:10:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:10:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:10:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:10:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:10:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:10:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:10:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:10:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:10:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:10:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:10:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:10:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:10:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:10:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:10:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:10:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:10:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:11:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:11:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:11:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:11:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:11:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:11:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:11:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:11:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:11:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:11:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:11:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:11:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:11:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:11:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:11:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:11:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:11:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:11:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:11:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:11:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:11:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:11:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:11:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:11:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:11:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:11:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:11:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:12:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:12:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:12:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:12:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:12:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:12:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:12:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:12:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:12:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:12:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:12:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:12:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:12:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:12:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:12:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:12:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:12:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:12:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:12:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:12:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:12:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:12:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:12:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:12:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:12:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:12:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:12:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:13:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:13:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:13:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:13:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:13:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:13:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:13:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:13:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:13:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:13:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:13:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:13:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:13:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:13:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:13:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:13:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:13:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:13:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:13:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:13:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:13:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:13:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:13:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:13:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:13:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:13:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:13:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:13:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:14:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:14:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:14:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:14:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:14:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:14:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:14:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:14:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:14:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:14:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:14:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:14:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:14:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:14:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:14:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:14:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:14:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:14:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:14:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:14:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:14:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:14:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:14:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:14:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:14:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:14:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:14:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:15:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:15:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:15:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:15:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:15:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:15:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:15:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:15:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:15:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:15:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:15:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:15:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:15:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:15:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:15:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:15:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:15:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:15:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:15:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:15:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:15:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:15:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:15:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:15:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:15:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:15:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:15:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:16:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:16:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:16:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:16:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:16:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:16:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:16:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:16:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:16:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:16:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:16:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:16:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:16:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:16:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:16:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:16:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:16:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:16:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:16:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:16:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:16:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:16:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:16:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:16:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:16:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:16:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:16:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:16:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:17:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:17:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:17:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:17:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:17:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:17:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:17:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:17:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:17:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:17:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:17:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:17:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:17:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:17:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:17:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:17:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:17:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:17:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:17:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:17:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:17:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:17:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:17:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:17:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:17:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:17:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:17:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:18:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:18:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:18:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:18:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:18:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:18:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:18:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:18:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:18:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:18:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:18:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:18:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:18:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:18:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:18:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:18:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:18:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:18:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:18:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:18:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:18:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:18:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:18:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:18:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:18:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:18:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:18:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:19:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:19:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:19:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:19:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:19:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:19:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:19:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:19:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:19:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:19:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:19:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:19:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:19:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:19:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:19:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:19:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:19:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:19:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:19:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:19:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:19:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:19:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:19:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:19:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:19:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:19:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:19:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:19:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:20:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:20:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:20:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:20:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:20:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:20:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:20:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:20:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:20:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:20:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:20:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:20:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:20:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:20:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:20:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:20:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:20:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:20:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:20:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:20:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:20:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:20:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:20:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:20:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:20:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:20:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:20:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:21:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:21:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:21:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:21:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:21:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:21:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:21:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:21:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:21:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:21:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:21:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:21:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:21:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:21:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:21:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:21:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:21:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:21:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:21:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:21:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:21:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:21:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:21:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:21:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:21:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:21:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:21:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:22:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:22:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:22:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:22:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:22:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:22:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:22:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:22:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:22:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:22:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:22:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:22:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:22:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:22:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:22:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:22:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:22:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:22:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:22:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:22:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:22:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:22:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:22:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:22:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:22:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:22:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:22:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:22:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:23:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:23:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:23:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:23:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:23:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:23:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:23:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:23:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:23:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:23:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:23:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:23:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:23:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:23:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:23:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:23:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:23:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:23:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:23:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:23:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:23:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:23:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:23:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:23:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:23:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:23:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:23:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:24:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:24:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:24:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:24:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:24:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:24:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:24:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:24:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:24:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:24:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:24:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:24:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:24:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:24:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:24:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:24:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:24:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:24:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:24:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:24:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:24:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:24:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:24:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:24:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:24:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:24:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:24:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:25:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:25:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:25:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:25:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:25:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:25:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:25:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:25:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:25:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:25:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:25:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:25:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:25:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:25:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:25:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:25:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:25:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:25:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:25:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:25:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:25:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:25:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:25:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:25:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:25:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:25:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:25:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:25:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:26:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:26:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:26:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:26:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:26:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:26:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:26:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:26:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:26:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:26:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:26:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:26:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:26:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:26:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:26:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:26:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:26:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:26:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:26:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:26:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:26:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:26:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:26:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:26:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:26:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:26:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:26:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:27:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:27:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:27:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:27:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:27:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:27:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:27:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:27:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:27:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:27:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:27:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:27:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:27:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:27:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:27:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:27:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:27:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:27:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:27:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:27:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:27:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:27:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:27:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:27:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:27:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:27:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:27:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:28:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:28:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:28:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:28:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:28:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:28:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:28:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:28:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:28:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:28:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:28:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:28:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:28:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:28:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:28:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:28:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:28:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:28:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:28:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:28:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:28:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:28:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:28:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:28:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:28:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:28:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:28:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:29:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:29:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:29:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:29:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:29:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:29:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:29:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:29:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:29:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:29:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:29:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:29:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:29:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:29:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:29:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:29:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:29:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:29:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:29:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:29:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:29:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:29:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:29:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:29:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:29:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:29:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:29:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:29:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:30:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:30:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:30:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:30:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:30:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:30:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:30:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:30:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:30:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:30:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:30:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:30:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:30:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:30:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:30:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:30:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:30:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:30:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:30:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:30:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:30:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:30:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:30:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:30:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:30:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:30:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:30:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:31:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:31:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:31:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:31:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:31:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:31:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:31:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:31:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:31:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:31:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:31:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:31:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:31:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:31:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:31:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:31:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:31:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:31:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:31:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:31:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:31:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:31:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:31:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:31:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:31:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:31:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:31:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:32:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  7 09:32:01 rdma06 dnf[571300]: Updating Subscription Management repositories.
Apr  7 09:32:01 rdma06 dnf[571300]: Unable to read consumer identity
Apr  7 09:32:01 rdma06 dnf[571300]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  7 09:32:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:32:03 rdma06 dnf[571300]: Copr repo for fmf owned by psss                 2.0 kB/s | 3.3 kB     00:01
Apr  7 09:32:03 rdma06 dnf[571300]: beaker-AppStream                                 89 kB/s | 3.2 kB     00:00
Apr  7 09:32:03 rdma06 dnf[571300]: beaker-AppStream-debuginfo                       33 kB/s | 1.5 kB     00:00
Apr  7 09:32:03 rdma06 dnf[571300]: beaker-BaseOS                                   124 kB/s | 2.7 kB     00:00
Apr  7 09:32:03 rdma06 dnf[571300]: beaker-BaseOS-debuginfo                          19 kB/s | 1.5 kB     00:00
Apr  7 09:32:03 rdma06 dnf[571300]: beaker-CRB                                       17 kB/s | 2.7 kB     00:00
Apr  7 09:32:03 rdma06 dnf[571300]: beaker-CRB-debuginfo                             18 kB/s | 1.5 kB     00:00
Apr  7 09:32:03 rdma06 dnf[571300]: beaker-HighAvailability                          23 kB/s | 2.8 kB     00:00
Apr  7 09:32:04 rdma06 dnf[571300]: beaker-HighAvailability-debuginfo               9.8 kB/s | 1.5 kB     00:00
Apr  7 09:32:04 rdma06 dnf[571300]: beaker-NFV                                       46 kB/s | 2.7 kB     00:00
Apr  7 09:32:04 rdma06 dnf[571300]: beaker-NFV-debuginfo                             23 kB/s | 1.5 kB     00:00
Apr  7 09:32:04 rdma06 dnf[571300]: beaker-RT                                        63 kB/s | 2.7 kB     00:00
Apr  7 09:32:04 rdma06 dnf[571300]: beaker-RT-debuginfo                              34 kB/s | 1.5 kB     00:00
Apr  7 09:32:04 rdma06 dnf[571300]: beaker-ResilientStorage                          68 kB/s | 2.8 kB     00:00
Apr  7 09:32:04 rdma06 dnf[571300]: beaker-ResilientStorage-debuginfo                65 kB/s | 1.5 kB     00:00
Apr  7 09:32:04 rdma06 dnf[571300]: beaker-SAP                                      132 kB/s | 2.7 kB     00:00
Apr  7 09:32:04 rdma06 dnf[571300]: beaker-SAP-debuginfo                             59 kB/s | 1.5 kB     00:00
Apr  7 09:32:04 rdma06 dnf[571300]: beaker-SAPHANA                                   64 kB/s | 2.7 kB     00:00
Apr  7 09:32:04 rdma06 dnf[571300]: beaker-SAPHANA-debuginfo                         42 kB/s | 1.5 kB     00:00
Apr  7 09:32:05 rdma06 dnf[571300]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr  7 09:32:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:32:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:32:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:32:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:32:05 rdma06 dnf[571300]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr  7 09:32:06 rdma06 dnf[571300]: Metadata cache created.
Apr  7 09:32:06 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  7 09:32:06 rdma06 systemd[1]: Finished dnf makecache.
Apr  7 09:32:06 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.408s CPU time.
Apr  7 09:32:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:32:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:32:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:32:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:32:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:32:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:32:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:32:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:32:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:32:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:32:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:32:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:32:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:32:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:32:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:32:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:32:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:32:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:32:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:32:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:32:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:32:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:32:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:33:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:33:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:33:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:33:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:33:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:33:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:33:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:33:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:33:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:33:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:33:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:33:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:33:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:33:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:33:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:33:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:33:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:33:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:33:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:33:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:33:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:33:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:33:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:33:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:33:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:33:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:33:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:34:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:34:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:34:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:34:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:34:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:34:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:34:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:34:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:34:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:34:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:34:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:34:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:34:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:34:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:34:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:34:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:34:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:34:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:34:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:34:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:34:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:34:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:34:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:34:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:34:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:34:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:34:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:35:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:35:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:35:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:35:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:35:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:35:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:35:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:35:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:35:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:35:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:35:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:35:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:35:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:35:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:35:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:35:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:35:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:35:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:35:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:35:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:35:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:35:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:35:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:35:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:35:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:35:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:35:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:35:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:36:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:36:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:36:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:36:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:36:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:36:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:36:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:36:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:36:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:36:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:36:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:36:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:36:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:36:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:36:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:36:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:36:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:36:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:36:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:36:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:36:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:36:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:36:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:36:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:36:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:36:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:36:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:37:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:37:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:37:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:37:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:37:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:37:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:37:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:37:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:37:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:37:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:37:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:37:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:37:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:37:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:37:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:37:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:37:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:37:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:37:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:37:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:37:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:37:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:37:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:37:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:37:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:37:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:37:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:38:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:38:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:38:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:38:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:38:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:38:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:38:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:38:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:38:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:38:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:38:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:38:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:38:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:38:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:38:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:38:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:38:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:38:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:38:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:38:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:38:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:38:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:38:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:38:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:38:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:38:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:38:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:38:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:39:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:39:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:39:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:39:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:39:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:39:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:39:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:39:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:39:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:39:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:39:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:39:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:39:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:39:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:39:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:39:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:39:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:39:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:39:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:39:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:39:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:39:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:39:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:39:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:39:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:39:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:39:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:40:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:40:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:40:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:40:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:40:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:40:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:40:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:40:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:40:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:40:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:40:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:40:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:40:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:40:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:40:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:40:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:40:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:40:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:40:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:40:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:40:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:40:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:40:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:40:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:40:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:40:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:40:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:41:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:41:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:41:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:41:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:41:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:41:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:41:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:41:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:41:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:41:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:41:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:41:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:41:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:41:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:41:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:41:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:41:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:41:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:41:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:41:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:41:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:41:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:41:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:41:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:41:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:41:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:41:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:41:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:42:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:42:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:42:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:42:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:42:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:42:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:42:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:42:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:42:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:42:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:42:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:42:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:42:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:42:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:42:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:42:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:42:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:42:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:42:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:42:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:42:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:42:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:42:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:42:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:42:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:42:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:42:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:43:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:43:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:43:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:43:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:43:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:43:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:43:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:43:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:43:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:43:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:43:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:43:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:43:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:43:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:43:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:43:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:43:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:43:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:43:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:43:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:43:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:43:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:43:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:43:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:43:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:43:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:43:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:44:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:44:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:44:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:44:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:44:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:44:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:44:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:44:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:44:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:44:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:44:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:44:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:44:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:44:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:44:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:44:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:44:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:44:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:44:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:44:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:44:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:44:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:44:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:44:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:44:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:44:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:44:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:45:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:45:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:45:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:45:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:45:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:45:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:45:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:45:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:45:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:45:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:45:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:45:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:45:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:45:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:45:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:45:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:45:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:45:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:45:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:45:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:45:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:45:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:45:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:45:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:45:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:45:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:45:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:45:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:46:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:46:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:46:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:46:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:46:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:46:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:46:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:46:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:46:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:46:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:46:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:46:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:46:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:46:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:46:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:46:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:46:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:46:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:46:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:46:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:46:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:46:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:46:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:46:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:46:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:46:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:46:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:47:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:47:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:47:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:47:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:47:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:47:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:47:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:47:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:47:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:47:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:47:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:47:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:47:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:47:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:47:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:47:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:47:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:47:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:47:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:47:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:47:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:47:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:47:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:47:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:47:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:47:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:47:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:48:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:48:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:48:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:48:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:48:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:48:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:48:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:48:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:48:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:48:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:48:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:48:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:48:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:48:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:48:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:48:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:48:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:48:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:48:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:48:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:48:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:48:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:48:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:48:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:48:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:48:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:48:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:48:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:49:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:49:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:49:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:49:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:49:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:49:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:49:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:49:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:49:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:49:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:49:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:49:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:49:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:49:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:49:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:49:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:49:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:49:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:49:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:49:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:49:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:49:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:49:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:49:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:49:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:49:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:49:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:50:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:50:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:50:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:50:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:50:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:50:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:50:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:50:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:50:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:50:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:50:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:50:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:50:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:50:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:50:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:50:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:50:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:50:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:50:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:50:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:50:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:50:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:50:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:50:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:50:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:50:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:50:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:51:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:51:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:51:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:51:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:51:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:51:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:51:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:51:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:51:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:51:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:51:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:51:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:51:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:51:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:51:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:51:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:51:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:51:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:51:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:51:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:51:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:51:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:51:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:51:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:51:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:51:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:51:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:51:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:52:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:52:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:52:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:52:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:52:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:52:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:52:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:52:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:52:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:52:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:52:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:52:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:52:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:52:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:52:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:52:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:52:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:52:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:52:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:52:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:52:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:52:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:52:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:52:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:52:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:52:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:52:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:53:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:53:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:53:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:53:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:53:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:53:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:53:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:53:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:53:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:53:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:53:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:53:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:53:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:53:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:53:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:53:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:53:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:53:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:53:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:53:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:53:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:53:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:53:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:53:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:53:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:53:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:53:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:54:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:54:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:54:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:54:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:54:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:54:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:54:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:54:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:54:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:54:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:54:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:54:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:54:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:54:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:54:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:54:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:54:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:54:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:54:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:54:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:54:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:54:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:54:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:54:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:54:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:54:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:54:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:54:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:55:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:55:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:55:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:55:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:55:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:55:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:55:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:55:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:55:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:55:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:55:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:55:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:55:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:55:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:55:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:55:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:55:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:55:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:55:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:55:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:55:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:55:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:55:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:55:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:55:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:55:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:55:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:56:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:56:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:56:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:56:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:56:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:56:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:56:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:56:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:56:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:56:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:56:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:56:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:56:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:56:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:56:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:56:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:56:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:56:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:56:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:56:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:56:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:56:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:56:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:56:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:56:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:56:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:56:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:57:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:57:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:57:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:57:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:57:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:57:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:57:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:57:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:57:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:57:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:57:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:57:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:57:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:57:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:57:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:57:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:57:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:57:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:57:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:57:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:57:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:57:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:57:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:57:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:57:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:57:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:57:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:57:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:58:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:58:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:58:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:58:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:58:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:58:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:58:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:58:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:58:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:58:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:58:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:58:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:58:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:58:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:58:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:58:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:58:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:58:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:58:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:58:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:58:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:58:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:58:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:58:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:58:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:58:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:58:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:59:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:59:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:59:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:59:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:59:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:59:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:59:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:59:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:59:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:59:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:59:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:59:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:59:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:59:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:59:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:59:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:59:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:59:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:59:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:59:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:59:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:59:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 09:59:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 09:59:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 09:59:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 09:59:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 09:59:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:00:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:00:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:00:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:00:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:00:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:00:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:00:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:00:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:00:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:00:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:00:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:00:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:00:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:00:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:00:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:00:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:00:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:00:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:00:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:00:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:00:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:00:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:00:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:00:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:00:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:00:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:00:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:01:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:01:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:01:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:01:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:01:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:01:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:01:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:01:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:01:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:01:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:01:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:01:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:01:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:01:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:01:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:01:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:01:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:01:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:01:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:01:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:01:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:01:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:01:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:01:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:01:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:01:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:01:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:01:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:02:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:02:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:02:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:02:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:02:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:02:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:02:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:02:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:02:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:02:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:02:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:02:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:02:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:02:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:02:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:02:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:02:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:02:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:02:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:02:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:02:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:02:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:02:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:02:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:02:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:02:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:02:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:03:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:03:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:03:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:03:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:03:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:03:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:03:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:03:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:03:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:03:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:03:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:03:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:03:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:03:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:03:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:03:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:03:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:03:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:03:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:03:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:03:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:03:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:03:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:03:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:03:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:03:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:03:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:04:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:04:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:04:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:04:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:04:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:04:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:04:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:04:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:04:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:04:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:04:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:04:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:04:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:04:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:04:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:04:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:04:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:04:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:04:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:04:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:04:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:04:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:04:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:04:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:04:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:04:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:04:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:04:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:05:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:05:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:05:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:05:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:05:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:05:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:05:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:05:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:05:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:05:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:05:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:05:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:05:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:05:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:05:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:05:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:05:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:05:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:05:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:05:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:05:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:05:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:05:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:05:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:05:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:05:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:05:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:06:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:06:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:06:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:06:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:06:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:06:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:06:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:06:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:06:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:06:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:06:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:06:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:06:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:06:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:06:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:06:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:06:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:06:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:06:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:06:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:06:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:06:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:06:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:06:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:06:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:06:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:06:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:07:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:07:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:07:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:07:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:07:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:07:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:07:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:07:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:07:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:07:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:07:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:07:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:07:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:07:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:07:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:07:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:07:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:07:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:07:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:07:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:07:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:07:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:07:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:07:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:07:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:07:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:07:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:07:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:08:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:08:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:08:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:08:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:08:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:08:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:08:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:08:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:08:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:08:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:08:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:08:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:08:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:08:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:08:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:08:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:08:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:08:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:08:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:08:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:08:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:08:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:08:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:08:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:08:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:08:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:08:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:09:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:09:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:09:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:09:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:09:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:09:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:09:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:09:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:09:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:09:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:09:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:09:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:09:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:09:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:09:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:09:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:09:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:09:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:09:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:09:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:09:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:09:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:09:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:09:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:09:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:09:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:09:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:10:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:10:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:10:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:10:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:10:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:10:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:10:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:10:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:10:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:10:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:10:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:10:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:10:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:10:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:10:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:10:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:10:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:10:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:10:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:10:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:10:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:10:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:10:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:10:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:10:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:10:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:10:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:10:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:11:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:11:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:11:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:11:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:11:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:11:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:11:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:11:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:11:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:11:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:11:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:11:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:11:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:11:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:11:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:11:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:11:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:11:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:11:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:11:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:11:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:11:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:11:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:11:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:11:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:11:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:11:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:12:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:12:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:12:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:12:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:12:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:12:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:12:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:12:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:12:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:12:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:12:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:12:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:12:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:12:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:12:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:12:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:12:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:12:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:12:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:12:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:12:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:12:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:12:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:12:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:12:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:12:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:12:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:13:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:13:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:13:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:13:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:13:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:13:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:13:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:13:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:13:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:13:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:13:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:13:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:13:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:13:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:13:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:13:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:13:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:13:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:13:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:13:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:13:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:13:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:13:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:13:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:13:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:13:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:13:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:13:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:14:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:14:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:14:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:14:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:14:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:14:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:14:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:14:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:14:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:14:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:14:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:14:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:14:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:14:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:14:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:14:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:14:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:14:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:14:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:14:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:14:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:14:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:14:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:14:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:14:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:14:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:14:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:15:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:15:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:15:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:15:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:15:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:15:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:15:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:15:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:15:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:15:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:15:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:15:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:15:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:15:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:15:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:15:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:15:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:15:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:15:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:15:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:15:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:15:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:15:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:15:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:15:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:15:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:15:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:16:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:16:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:16:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:16:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:16:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:16:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:16:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:16:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:16:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:16:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:16:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:16:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:16:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:16:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:16:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:16:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:16:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:16:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:16:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:16:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:16:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:16:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:16:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:16:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:16:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:16:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:16:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:16:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:17:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:17:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:17:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:17:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:17:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:17:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:17:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:17:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:17:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:17:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:17:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:17:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:17:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:17:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:17:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:17:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:17:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:17:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:17:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:17:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:17:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:17:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:17:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:17:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:17:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:17:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:17:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:18:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:18:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:18:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:18:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:18:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:18:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:18:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:18:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:18:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:18:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:18:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:18:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:18:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:18:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:18:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:18:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:18:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:18:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:18:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:18:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:18:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:18:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:18:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:18:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:18:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:18:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:18:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:19:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:19:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:19:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:19:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:19:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:19:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:19:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:19:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:19:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:19:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:19:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:19:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:19:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:19:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:19:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:19:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:19:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:19:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:19:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:19:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:19:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:19:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:19:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:19:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:19:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:19:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:19:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:20:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:20:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:20:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:20:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:20:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:20:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:20:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:20:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:20:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:20:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:20:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:20:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:20:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:20:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:20:32 rdma06 NetworkManager[1025]: <info>  [1680877232.8173] dhcp4 (eno1): state changed new lease, address=10.73.131.45
Apr  7 10:20:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:20:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:20:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:20:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:20:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:20:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:20:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:20:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:20:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:20:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:20:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:20:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:20:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:20:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:21:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:21:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:21:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:21:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:21:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:21:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:21:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:21:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:21:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:21:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:21:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:21:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:21:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:21:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:21:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:21:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:21:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:21:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:21:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:21:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:21:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:21:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:21:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:21:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:21:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:21:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:21:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:22:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:22:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:22:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:22:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:22:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:22:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:22:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:22:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:22:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:22:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:22:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:22:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:22:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:22:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:22:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:22:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:22:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:22:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:22:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:22:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:22:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:22:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:22:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:22:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:22:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:22:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:22:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:23:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:23:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:23:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:23:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:23:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:23:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:23:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:23:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:23:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:23:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:23:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:23:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:23:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:23:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:23:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:23:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:23:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:23:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:23:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:23:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:23:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:23:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:23:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:23:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:23:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:23:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:23:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:23:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:24:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:24:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:24:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:24:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:24:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:24:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:24:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:24:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:24:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:24:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:24:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:24:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:24:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:24:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:24:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:24:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:24:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:24:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:24:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:24:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:24:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:24:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:24:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:24:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:24:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:24:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:24:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:25:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:25:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:25:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:25:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:25:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:25:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:25:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:25:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:25:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:25:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:25:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:25:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:25:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:25:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:25:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:25:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:25:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:25:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:25:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:25:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:25:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:25:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:25:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:25:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:25:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:25:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:25:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:26:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:26:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:26:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:26:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:26:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:26:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:26:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:26:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:26:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:26:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:26:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:26:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:26:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:26:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:26:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:26:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:26:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:26:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:26:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:26:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:26:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:26:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:26:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:26:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:26:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:26:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:26:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:26:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:27:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:27:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:27:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:27:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:27:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:27:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:27:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:27:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:27:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:27:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:27:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:27:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:27:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:27:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:27:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:27:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:27:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:27:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:27:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:27:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:27:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:27:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:27:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:27:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:27:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:27:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:27:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:28:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:28:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:28:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:28:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:28:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:28:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:28:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:28:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:28:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:28:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:28:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:28:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:28:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:28:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:28:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:28:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:28:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:28:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:28:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:28:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:28:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:28:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:28:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:28:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:28:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:28:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:28:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:29:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:29:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:29:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:29:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:29:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:29:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:29:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:29:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:29:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:29:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:29:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:29:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:29:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:29:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:29:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:29:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:29:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:29:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:29:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:29:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:29:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:29:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:29:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:29:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:29:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:29:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:29:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:29:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:30:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:30:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:30:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:30:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:30:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:30:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:30:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:30:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:30:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:30:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:30:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:30:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:30:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:30:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:30:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:30:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:30:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:30:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:30:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:30:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:30:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:30:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:30:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:30:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:30:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:30:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:30:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:31:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:31:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:31:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:31:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:31:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:31:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:31:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:31:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:31:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:31:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:31:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:31:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:31:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:31:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:31:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:31:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:31:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:31:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:31:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:31:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:31:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:31:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:31:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:31:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:31:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:31:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:31:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:32:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:32:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:32:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:32:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:32:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:32:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:32:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:32:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:32:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:32:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:32:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:32:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:32:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:32:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:32:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:32:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:32:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:32:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:32:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:32:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:32:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:32:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:32:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:32:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:32:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:32:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:32:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:32:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:33:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:33:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:33:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:33:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:33:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:33:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:33:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:33:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:33:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:33:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:33:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:33:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:33:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:33:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:33:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:33:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:33:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:33:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:33:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:33:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:33:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:33:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:33:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:33:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:33:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:33:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:33:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:34:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:34:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:34:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:34:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:34:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:34:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:34:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:34:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:34:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:34:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:34:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:34:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:34:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:34:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:34:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:34:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:34:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:34:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:34:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:34:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:34:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:34:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:34:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:34:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:34:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:34:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:34:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:35:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:35:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:35:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:35:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:35:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:35:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:35:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:35:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:35:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:35:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:35:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:35:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:35:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:35:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:35:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:35:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:35:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:35:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:35:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:35:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:35:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:35:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:35:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:35:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:35:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:35:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:35:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:36:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:36:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:36:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:36:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:36:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:36:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:36:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:36:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:36:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:36:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:36:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:36:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:36:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:36:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:36:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:36:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:36:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:36:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:36:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:36:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:36:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:36:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:36:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:36:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:36:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:36:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:36:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:36:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:37:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:37:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:37:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:37:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:37:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:37:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:37:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:37:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:37:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:37:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:37:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:37:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:37:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:37:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:37:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:37:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:37:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:37:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:37:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:37:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:37:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:37:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:37:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:37:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:37:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:37:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:37:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:38:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:38:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:38:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:38:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:38:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:38:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:38:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:38:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:38:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:38:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:38:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:38:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:38:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:38:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:38:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:38:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:38:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:38:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:38:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:38:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:38:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:38:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:38:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:38:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:38:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:38:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:38:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:39:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:39:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:39:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:39:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:39:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:39:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:39:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:39:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:39:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:39:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:39:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:39:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:39:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:39:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:39:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:39:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:39:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:39:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:39:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:39:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:39:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:39:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:39:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:39:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:39:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:39:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:39:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:39:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:40:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:40:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:40:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:40:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:40:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:40:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:40:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:40:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:40:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:40:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:40:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:40:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:40:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:40:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:40:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:40:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:40:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:40:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:40:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:40:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:40:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:40:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:40:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:40:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:40:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:40:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:40:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:41:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  7 10:41:01 rdma06 dnf[574111]: Updating Subscription Management repositories.
Apr  7 10:41:01 rdma06 dnf[574111]: Unable to read consumer identity
Apr  7 10:41:01 rdma06 dnf[574111]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  7 10:41:01 rdma06 dnf[574111]: Metadata cache refreshed recently.
Apr  7 10:41:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  7 10:41:01 rdma06 systemd[1]: Finished dnf makecache.
Apr  7 10:41:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:41:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:41:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:41:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:41:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:41:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:41:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:41:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:41:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:41:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:41:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:41:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:41:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:41:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:41:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:41:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:41:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:41:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:41:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:41:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:41:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:41:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:41:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:41:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:41:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:41:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:41:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:42:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:42:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:42:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:42:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:42:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:42:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:42:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:42:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:42:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:42:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:42:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:42:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:42:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:42:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:42:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:42:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:42:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:42:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:42:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:42:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:42:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:42:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:42:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:42:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:42:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:42:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:42:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:42:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:43:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:43:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:43:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:43:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:43:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:43:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:43:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:43:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:43:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:43:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:43:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:43:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:43:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:43:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:43:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:43:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:43:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:43:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:43:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:43:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:43:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:43:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:43:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:43:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:43:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:43:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:43:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:44:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:44:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:44:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:44:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:44:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:44:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:44:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:44:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:44:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:44:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:44:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:44:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:44:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:44:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:44:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:44:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:44:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:44:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:44:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:44:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:44:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:44:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:44:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:44:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:44:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:44:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:44:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:45:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:45:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:45:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:45:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:45:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:45:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:45:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:45:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:45:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:45:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:45:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:45:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:45:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:45:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:45:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:45:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:45:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:45:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:45:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:45:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:45:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:45:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:45:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:45:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:45:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:45:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:45:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:45:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:46:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:46:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:46:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:46:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:46:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:46:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:46:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:46:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:46:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:46:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:46:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:46:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:46:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:46:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:46:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:46:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:46:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:46:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:46:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:46:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:46:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:46:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:46:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:46:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:46:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:46:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:46:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:47:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:47:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:47:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:47:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:47:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:47:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:47:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:47:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:47:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:47:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:47:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:47:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:47:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:47:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:47:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:47:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:47:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:47:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:47:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:47:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:47:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:47:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:47:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:47:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:47:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:47:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:47:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:48:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:48:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:48:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:48:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:48:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:48:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:48:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:48:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:48:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:48:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:48:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:48:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:48:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:48:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:48:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:48:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:48:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:48:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:48:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:48:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:48:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:48:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:48:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:48:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:48:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:48:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:48:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:48:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:49:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:49:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:49:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:49:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:49:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:49:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:49:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:49:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:49:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:49:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:49:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:49:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:49:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:49:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:49:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:49:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:49:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:49:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:49:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:49:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:49:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:49:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:49:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:49:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:49:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:49:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:49:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:50:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:50:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:50:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:50:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:50:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:50:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:50:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:50:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:50:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:50:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:50:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:50:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:50:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:50:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:50:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:50:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:50:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:50:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:50:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:50:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:50:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:50:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:50:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:50:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:50:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:50:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:50:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:51:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:51:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:51:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:51:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:51:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:51:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:51:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:51:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:51:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:51:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:51:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:51:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:51:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:51:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:51:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:51:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:51:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:51:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:51:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:51:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:51:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:51:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:51:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:51:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:51:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:51:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:51:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:52:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:52:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:52:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:52:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:52:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:52:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:52:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:52:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:52:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:52:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:52:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:52:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:52:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:52:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:52:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:52:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:52:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:52:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:52:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:52:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:52:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:52:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:52:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:52:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:52:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:52:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:52:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:52:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:53:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:53:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:53:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:53:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:53:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:53:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:53:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:53:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:53:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:53:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:53:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:53:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:53:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:53:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:53:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:53:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:53:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:53:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:53:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:53:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:53:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:53:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:53:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:53:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:53:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:53:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:53:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:54:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:54:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:54:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:54:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:54:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:54:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:54:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:54:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:54:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:54:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:54:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:54:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:54:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:54:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:54:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:54:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:54:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:54:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:54:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:54:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:54:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:54:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:54:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:54:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:54:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:54:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:54:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:55:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:55:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:55:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:55:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:55:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:55:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:55:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:55:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:55:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:55:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:55:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:55:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:55:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:55:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:55:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:55:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:55:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:55:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:55:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:55:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:55:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:55:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:55:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:55:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:55:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:55:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:55:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:55:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:56:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:56:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:56:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:56:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:56:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:56:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:56:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:56:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:56:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:56:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:56:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:56:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:56:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:56:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:56:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:56:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:56:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:56:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:56:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:56:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:56:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:56:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:56:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:56:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:56:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:56:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:56:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:57:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:57:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:57:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:57:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:57:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:57:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:57:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:57:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:57:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:57:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:57:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:57:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:57:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:57:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:57:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:57:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:57:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:57:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:57:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:57:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:57:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:57:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:57:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:57:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:57:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:57:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:57:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:58:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:58:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:58:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:58:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:58:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:58:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:58:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:58:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:58:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:58:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:58:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:58:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:58:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:58:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:58:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:58:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:58:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:58:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:58:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:58:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:58:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:58:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:58:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:58:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:58:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:58:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:58:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:58:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:59:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:59:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:59:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:59:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:59:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:59:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:59:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:59:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:59:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:59:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:59:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:59:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:59:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:59:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:59:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:59:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:59:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:59:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:59:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:59:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:59:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:59:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 10:59:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 10:59:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 10:59:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 10:59:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 10:59:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:00:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:00:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:00:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:00:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:00:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:00:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:00:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:00:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:00:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:00:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:00:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:00:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:00:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:00:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:00:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:00:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:00:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:00:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:00:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:00:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:00:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:00:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:00:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:00:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:00:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:00:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:00:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:01:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:01:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:01:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:01:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:01:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:01:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:01:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:01:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:01:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:01:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:01:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:01:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:01:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:01:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:01:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:01:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:01:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:01:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:01:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:01:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:01:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:01:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:01:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:01:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:01:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:01:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:01:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:01:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:02:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:02:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:02:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:02:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:02:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:02:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:02:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:02:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:02:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:02:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:02:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:02:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:02:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:02:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:02:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:02:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:02:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:02:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:02:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:02:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:02:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:02:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:02:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:02:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:02:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:02:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:02:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:03:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:03:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:03:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:03:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:03:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:03:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:03:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:03:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:03:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:03:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:03:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:03:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:03:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:03:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:03:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:03:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:03:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:03:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:03:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:03:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:03:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:03:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:03:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:03:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:03:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:03:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:03:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:04:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:04:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:04:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:04:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:04:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:04:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:04:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:04:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:04:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:04:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:04:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:04:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:04:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:04:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:04:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:04:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:04:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:04:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:04:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:04:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:04:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:04:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:04:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:04:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:04:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:04:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:04:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:04:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:05:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:05:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:05:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:05:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:05:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:05:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:05:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:05:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:05:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:05:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:05:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:05:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:05:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:05:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:05:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:05:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:05:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:05:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:05:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:05:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:05:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:05:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:05:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:05:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:05:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:05:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:05:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:06:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:06:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:06:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:06:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:06:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:06:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:06:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:06:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:06:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:06:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:06:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:06:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:06:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:06:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:06:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:06:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:06:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:06:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:06:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:06:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:06:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:06:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:06:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:06:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:06:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:06:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:06:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:07:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:07:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:07:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:07:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:07:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:07:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:07:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:07:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:07:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:07:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:07:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:07:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:07:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:07:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:07:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:07:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:07:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:07:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:07:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:07:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:07:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:07:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:07:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:07:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:07:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:07:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:07:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:08:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:08:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:08:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:08:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:08:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:08:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:08:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:08:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:08:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:08:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:08:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:08:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:08:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:08:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:08:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:08:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:08:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:08:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:08:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:08:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:08:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:08:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:08:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:08:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:08:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:08:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:08:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:08:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:09:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:09:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:09:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:09:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:09:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:09:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:09:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:09:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:09:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:09:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:09:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:09:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:09:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:09:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:09:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:09:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:09:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:09:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:09:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:09:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:09:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:09:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:09:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:09:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:09:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:09:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:09:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:10:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:10:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:10:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:10:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:10:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:10:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:10:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:10:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:10:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:10:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:10:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:10:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:10:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:10:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:10:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:10:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:10:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:10:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:10:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:10:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:10:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:10:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:10:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:10:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:10:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:10:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:10:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:11:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:11:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:11:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:11:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:11:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:11:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:11:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:11:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:11:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:11:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:11:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:11:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:11:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:11:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:11:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:11:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:11:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:11:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:11:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:11:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:11:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:11:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:11:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:11:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:11:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:11:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:11:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:11:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:12:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:12:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:12:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:12:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:12:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:12:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:12:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:12:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:12:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:12:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:12:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:12:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:12:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:12:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:12:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:12:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:12:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:12:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:12:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:12:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:12:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:12:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:12:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:12:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:12:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:12:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:12:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:13:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:13:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:13:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:13:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:13:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:13:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:13:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:13:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:13:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:13:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:13:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:13:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:13:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:13:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:13:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:13:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:13:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:13:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:13:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:13:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:13:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:13:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:13:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:13:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:13:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:13:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:13:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:14:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:14:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:14:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:14:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:14:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:14:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:14:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:14:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:14:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:14:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:14:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:14:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:14:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:14:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:14:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:14:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:14:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:14:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:14:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:14:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:14:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:14:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:14:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:14:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:14:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:14:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:14:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:14:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:15:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:15:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:15:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:15:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:15:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:15:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:15:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:15:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:15:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:15:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:15:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:15:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:15:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:15:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:15:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:15:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:15:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:15:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:15:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:15:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:15:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:15:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:15:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:15:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:15:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:15:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:15:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:16:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:16:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:16:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:16:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:16:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:16:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:16:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:16:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:16:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:16:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:16:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:16:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:16:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:16:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:16:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:16:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:16:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:16:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:16:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:16:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:16:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:16:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:16:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:16:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:16:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:16:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:16:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:17:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:17:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:17:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:17:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:17:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:17:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:17:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:17:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:17:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:17:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:17:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:17:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:17:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:17:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:17:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:17:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:17:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:17:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:17:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:17:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:17:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:17:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:17:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:17:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:17:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:17:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:17:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:17:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:18:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:18:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:18:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:18:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:18:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:18:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:18:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:18:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:18:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:18:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:18:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:18:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:18:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:18:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:18:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:18:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:18:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:18:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:18:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:18:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:18:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:18:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:18:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:18:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:18:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:18:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:18:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:19:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:19:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:19:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:19:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:19:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:19:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:19:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:19:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:19:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:19:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:19:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:19:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:19:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:19:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:19:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:19:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:19:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:19:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:19:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:19:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:19:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:19:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:19:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:19:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:19:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:19:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:19:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:20:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:20:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:20:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:20:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:20:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:20:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:20:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:20:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:20:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:20:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:20:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:20:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:20:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:20:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:20:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:20:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:20:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:20:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:20:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:20:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:20:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:20:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:20:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:20:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:20:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:20:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:20:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:20:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:21:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:21:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:21:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:21:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:21:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:21:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:21:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:21:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:21:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:21:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:21:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:21:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:21:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:21:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:21:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:21:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:21:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:21:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:21:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:21:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:21:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:21:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:21:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:21:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:21:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:21:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:21:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:22:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:22:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:22:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:22:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:22:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:22:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:22:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:22:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:22:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:22:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:22:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:22:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:22:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:22:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:22:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:22:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:22:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:22:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:22:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:22:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:22:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:22:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:22:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:22:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:22:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:22:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:22:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:23:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:23:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:23:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:23:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:23:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:23:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:23:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:23:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:23:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:23:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:23:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:23:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:23:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:23:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:23:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:23:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:23:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:23:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:23:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:23:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:23:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:23:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:23:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:23:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:23:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:23:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:23:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:23:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:24:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:24:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:24:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:24:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:24:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:24:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:24:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:24:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:24:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:24:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:24:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:24:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:24:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:24:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:24:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:24:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:24:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:24:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:24:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:24:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:24:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:24:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:24:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:24:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:24:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:24:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:24:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:25:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:25:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:25:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:25:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:25:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:25:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:25:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:25:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:25:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:25:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:25:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:25:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:25:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:25:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:25:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:25:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:25:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:25:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:25:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:25:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:25:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:25:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:25:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:25:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:25:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:25:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:25:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:26:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:26:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:26:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:26:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:26:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:26:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:26:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:26:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:26:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:26:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:26:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:26:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:26:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:26:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:26:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:26:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:26:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:26:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:26:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:26:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:26:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:26:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:26:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:26:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:26:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:26:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:26:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:27:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:27:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:27:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:27:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:27:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:27:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:27:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:27:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:27:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:27:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:27:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:27:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:27:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:27:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:27:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:27:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:27:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:27:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:27:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:27:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:27:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:27:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:27:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:27:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:27:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:27:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:27:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:27:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:28:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:28:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:28:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:28:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:28:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:28:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:28:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:28:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:28:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:28:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:28:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:28:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:28:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:28:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:28:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:28:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:28:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:28:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:28:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:28:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:28:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:28:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:28:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:28:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:28:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:28:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:28:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:29:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:29:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:29:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:29:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:29:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:29:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:29:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:29:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:29:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:29:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:29:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:29:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:29:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:29:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:29:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:29:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:29:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:29:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:29:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:29:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:29:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:29:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:29:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:29:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:29:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:29:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:29:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:30:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:30:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:30:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:30:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:30:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:30:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:30:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:30:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:30:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:30:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:30:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:30:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:30:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:30:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:30:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:30:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:30:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:30:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:30:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:30:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:30:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:30:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:30:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:30:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:30:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:30:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:30:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:30:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:31:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:31:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:31:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:31:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:31:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:31:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:31:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:31:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:31:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:31:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:31:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:31:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:31:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:31:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:31:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:31:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:31:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:31:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:31:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:31:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:31:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:31:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:31:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:31:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:31:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:31:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:31:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:32:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:32:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:32:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:32:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:32:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:32:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:32:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:32:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:32:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:32:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:32:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:32:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:32:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:32:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:32:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:32:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:32:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:32:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:32:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:32:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:32:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:32:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:32:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:32:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:32:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:32:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:32:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:33:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:33:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:33:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:33:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:33:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:33:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:33:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:33:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:33:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:33:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:33:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:33:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:33:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:33:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:33:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:33:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:33:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:33:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:33:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:33:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:33:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:33:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:33:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:33:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:33:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:33:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:33:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:33:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:34:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:34:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:34:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:34:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:34:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:34:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:34:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:34:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:34:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:34:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:34:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:34:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:34:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:34:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:34:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:34:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:34:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:34:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:34:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:34:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:34:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:34:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:34:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:34:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:34:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:34:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:34:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:35:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:35:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:35:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:35:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:35:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:35:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:35:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:35:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:35:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:35:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:35:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:35:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:35:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:35:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:35:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:35:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:35:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:35:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:35:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:35:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:35:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:35:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:35:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:35:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:35:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:35:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:35:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:36:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:36:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:36:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:36:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:36:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:36:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:36:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:36:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:36:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:36:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:36:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:36:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:36:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:36:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:36:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:36:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:36:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:36:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:36:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:36:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:36:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:36:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:36:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:36:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:36:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:36:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:36:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:36:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:37:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:37:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:37:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:37:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:37:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:37:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:37:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:37:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:37:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:37:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:37:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:37:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:37:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:37:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:37:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:37:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:37:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:37:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:37:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:37:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:37:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:37:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:37:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:37:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:37:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:37:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:37:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:38:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:38:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:38:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:38:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:38:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:38:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:38:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:38:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:38:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:38:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:38:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:38:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:38:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:38:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:38:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:38:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:38:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:38:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:38:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:38:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:38:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:38:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:38:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:38:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:38:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:38:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:38:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:39:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:39:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:39:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:39:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:39:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:39:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:39:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:39:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:39:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:39:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:39:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:39:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:39:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:39:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:39:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:39:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:39:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:39:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:39:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:39:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:39:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:39:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:39:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:39:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:39:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:39:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:39:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:39:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:40:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:40:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:40:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:40:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:40:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:40:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:40:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:40:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:40:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:40:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:40:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:40:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:40:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:40:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:40:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:40:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:40:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:40:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:40:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:40:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:40:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:40:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:40:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:40:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:40:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:40:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:40:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:41:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:41:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:41:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:41:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:41:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:41:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:41:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:41:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:41:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:41:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:41:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:41:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:41:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:41:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:41:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:41:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:41:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:41:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:41:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:41:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:41:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:41:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:41:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:41:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:41:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:41:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:41:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:42:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:42:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:42:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:42:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:42:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:42:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:42:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:42:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:42:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:42:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:42:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:42:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:42:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:42:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:42:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:42:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:42:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:42:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:42:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:42:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:42:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:42:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:42:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:42:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:42:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:42:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:42:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:43:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:43:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:43:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:43:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:43:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:43:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:43:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:43:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:43:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:43:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:43:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:43:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:43:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:43:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:43:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:43:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:43:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:43:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:43:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:43:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:43:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:43:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:43:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:43:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:43:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:43:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:43:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:43:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:44:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:44:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:44:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:44:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:44:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:44:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:44:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:44:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:44:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:44:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:44:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:44:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:44:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:44:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:44:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:44:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:44:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:44:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:44:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:44:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:44:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:44:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:44:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:44:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:44:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:44:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:44:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:45:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:45:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:45:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:45:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:45:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:45:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:45:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:45:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:45:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:45:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:45:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:45:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:45:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:45:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:45:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:45:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:45:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:45:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:45:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:45:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:45:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:45:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:45:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:45:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:45:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:45:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:45:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:46:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:46:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:46:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:46:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:46:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:46:11 rdma06 chronyd[1034]: Selected source 10.11.160.238
Apr  7 11:46:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:46:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:46:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:46:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:46:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:46:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:46:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:46:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:46:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:46:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:46:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:46:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:46:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:46:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:46:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:46:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:46:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:46:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:46:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:46:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:46:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:46:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:46:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:47:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:47:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:47:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:47:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:47:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:47:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:47:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:47:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:47:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:47:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:47:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:47:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:47:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:47:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:47:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:47:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:47:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:47:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:47:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:47:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:47:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:47:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:47:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:47:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:47:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:47:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:47:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:48:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:48:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:48:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:48:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:48:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:48:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:48:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:48:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:48:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:48:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:48:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:48:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:48:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:48:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:48:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:48:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:48:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:48:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:48:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:48:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:48:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:48:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:48:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:48:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:48:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:48:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:48:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:49:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:49:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:49:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:49:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:49:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:49:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:49:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:49:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:49:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:49:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:49:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:49:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:49:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:49:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:49:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:49:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:49:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:49:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:49:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:49:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:49:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:49:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:49:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:49:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:49:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:49:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:49:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:49:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:50:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:50:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:50:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:50:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:50:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:50:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:50:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:50:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:50:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:50:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:50:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:50:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:50:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:50:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:50:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:50:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:50:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:50:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:50:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:50:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:50:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:50:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:50:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:50:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:50:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:50:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:50:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:51:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:51:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:51:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:51:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:51:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:51:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:51:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:51:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:51:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:51:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:51:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:51:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:51:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:51:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:51:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:51:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:51:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:51:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:51:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:51:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:51:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:51:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:51:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:51:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:51:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:51:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:51:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:52:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:52:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:52:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:52:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:52:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:52:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:52:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:52:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:52:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:52:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:52:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:52:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:52:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:52:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:52:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:52:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:52:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:52:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:52:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:52:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:52:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:52:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:52:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:52:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:52:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:52:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:52:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:52:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:53:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:53:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:53:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:53:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:53:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:53:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:53:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:53:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:53:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:53:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:53:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:53:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:53:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:53:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:53:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:53:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:53:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:53:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:53:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:53:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:53:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:53:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:53:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:53:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:53:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:53:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:53:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:54:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:54:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:54:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:54:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:54:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:54:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:54:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:54:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:54:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:54:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:54:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:54:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:54:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:54:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:54:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:54:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:54:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:54:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:54:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:54:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:54:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:54:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:54:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:54:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:54:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:54:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:54:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:55:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:55:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:55:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:55:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:55:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:55:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:55:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:55:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:55:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:55:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:55:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:55:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:55:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:55:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:55:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:55:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:55:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:55:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:55:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:55:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:55:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:55:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:55:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:55:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:55:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:55:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:55:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:55:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:56:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:56:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:56:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:56:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:56:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:56:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:56:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:56:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:56:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:56:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:56:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:56:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:56:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:56:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:56:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:56:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:56:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:56:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:56:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:56:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:56:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:56:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:56:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:56:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:56:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:56:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:56:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:57:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:57:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:57:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:57:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:57:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:57:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:57:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:57:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:57:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:57:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:57:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:57:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:57:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:57:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:57:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:57:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:57:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:57:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:57:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:57:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:57:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:57:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:57:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:57:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:57:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:57:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:57:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:58:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:58:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:58:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:58:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:58:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:58:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:58:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:58:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:58:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:58:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:58:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:58:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:58:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:58:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:58:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:58:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:58:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:58:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:58:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:58:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:58:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:58:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:58:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:58:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:58:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:58:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:58:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:59:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:59:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:59:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:59:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:59:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:59:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:59:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:59:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:59:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:59:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:59:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:59:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:59:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:59:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:59:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:59:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:59:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:59:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:59:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:59:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:59:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:59:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:59:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 11:59:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 11:59:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 11:59:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 11:59:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 11:59:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:00:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:00:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:00:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:00:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:00:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:00:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:00:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:00:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:00:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:00:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:00:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:00:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:00:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:00:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:00:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:00:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:00:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:00:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:00:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:00:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:00:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:00:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:00:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:00:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:00:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:00:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:00:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:01:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:01:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:01:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:01:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:01:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:01:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:01:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:01:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:01:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:01:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:01:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:01:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:01:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:01:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:01:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:01:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:01:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:01:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:01:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:01:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:01:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:01:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:01:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:01:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:01:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:01:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:01:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:02:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:02:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:02:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:02:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:02:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:02:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:02:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:02:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:02:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:02:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:02:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:02:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:02:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:02:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:02:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:02:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:02:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:02:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:02:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:02:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:02:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:02:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:02:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:02:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:02:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:02:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:02:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:02:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:03:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:03:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:03:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:03:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:03:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:03:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:03:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:03:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:03:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:03:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:03:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:03:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:03:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:03:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:03:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:03:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:03:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:03:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:03:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:03:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:03:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:03:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:03:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:03:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:03:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:03:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:03:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:04:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:04:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:04:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:04:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:04:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:04:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:04:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:04:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:04:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:04:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:04:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:04:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:04:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:04:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:04:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:04:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:04:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:04:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:04:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:04:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:04:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:04:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:04:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:04:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:04:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:04:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:04:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:05:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:05:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:05:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:05:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:05:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:05:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:05:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:05:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:05:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:05:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:05:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:05:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:05:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:05:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:05:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:05:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:05:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:05:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:05:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:05:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:05:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:05:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:05:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:05:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:05:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:05:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:05:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:05:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:06:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:06:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:06:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:06:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:06:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:06:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:06:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:06:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:06:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:06:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:06:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:06:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:06:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:06:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:06:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:06:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:06:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:06:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:06:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:06:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:06:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:06:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:06:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:06:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:06:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:06:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:06:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:07:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:07:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:07:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:07:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:07:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:07:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:07:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:07:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:07:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:07:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:07:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:07:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:07:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:07:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:07:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:07:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:07:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:07:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:07:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:07:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:07:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:07:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:07:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:07:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:07:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:07:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:07:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:08:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:08:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:08:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:08:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:08:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:08:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:08:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:08:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:08:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:08:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:08:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:08:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:08:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:08:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:08:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:08:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:08:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:08:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:08:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:08:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:08:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:08:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:08:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:08:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:08:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:08:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:08:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:08:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:09:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:09:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:09:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:09:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:09:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:09:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:09:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:09:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:09:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:09:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:09:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:09:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:09:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:09:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:09:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:09:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:09:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:09:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:09:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:09:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:09:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:09:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:09:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:09:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:09:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:09:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:09:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:10:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:10:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:10:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:10:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:10:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:10:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:10:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:10:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:10:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:10:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:10:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:10:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:10:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:10:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:10:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:10:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:10:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:10:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:10:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:10:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:10:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:10:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:10:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:10:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:10:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:10:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:10:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:11:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:11:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:11:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:11:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:11:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:11:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:11:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:11:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:11:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:11:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:11:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:11:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:11:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:11:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:11:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:11:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:11:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:11:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:11:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:11:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:11:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:11:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:11:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:11:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:11:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:11:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:11:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:11:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:12:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:12:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:12:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:12:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:12:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:12:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:12:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:12:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:12:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:12:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:12:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:12:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:12:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:12:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:12:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:12:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:12:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:12:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:12:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:12:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:12:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:12:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:12:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:12:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:12:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:12:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:12:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:13:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:13:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:13:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:13:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:13:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:13:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:13:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:13:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:13:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:13:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:13:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:13:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:13:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:13:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:13:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:13:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:13:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:13:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:13:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:13:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:13:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:13:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:13:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:13:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:13:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:13:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:13:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:14:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  7 12:14:01 rdma06 dnf[577877]: Updating Subscription Management repositories.
Apr  7 12:14:01 rdma06 dnf[577877]: Unable to read consumer identity
Apr  7 12:14:01 rdma06 dnf[577877]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  7 12:14:01 rdma06 dnf[577877]: Metadata cache refreshed recently.
Apr  7 12:14:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  7 12:14:01 rdma06 systemd[1]: Finished dnf makecache.
Apr  7 12:14:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:14:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:14:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:14:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:14:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:14:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:14:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:14:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:14:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:14:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:14:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:14:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:14:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:14:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:14:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:14:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:14:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:14:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:14:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:14:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:14:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:14:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:14:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:14:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:14:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:14:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:14:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:15:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:15:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:15:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:15:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:15:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:15:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:15:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:15:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:15:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:15:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:15:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:15:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:15:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:15:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:15:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:15:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:15:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:15:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:15:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:15:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:15:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:15:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:15:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:15:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:15:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:15:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:15:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:15:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:16:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:16:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:16:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:16:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:16:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:16:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:16:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:16:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:16:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:16:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:16:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:16:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:16:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:16:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:16:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:16:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:16:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:16:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:16:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:16:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:16:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:16:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:16:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:16:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:16:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:16:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:16:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:17:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:17:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:17:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:17:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:17:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:17:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:17:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:17:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:17:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:17:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:17:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:17:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:17:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:17:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:17:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:17:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:17:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:17:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:17:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:17:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:17:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:17:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:17:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:17:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:17:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:17:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:17:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:18:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:18:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:18:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:18:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:18:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:18:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:18:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:18:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:18:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:18:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:18:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:18:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:18:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:18:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:18:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:18:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:18:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:18:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:18:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:18:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:18:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:18:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:18:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:18:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:18:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:18:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:18:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:18:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:19:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:19:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:19:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:19:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:19:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:19:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:19:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:19:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:19:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:19:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:19:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:19:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:19:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:19:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:19:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:19:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:19:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:19:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:19:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:19:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:19:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:19:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:19:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:19:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:19:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:19:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:19:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:20:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:20:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:20:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:20:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:20:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:20:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:20:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:20:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:20:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:20:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:20:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:20:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:20:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:20:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:20:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:20:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:20:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:20:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:20:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:20:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:20:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:20:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:20:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:20:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:20:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:20:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:20:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:21:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:21:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:21:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:21:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:21:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:21:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:21:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:21:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:21:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:21:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:21:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:21:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:21:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:21:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:21:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:21:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:21:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:21:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:21:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:21:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:21:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:21:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:21:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:21:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:21:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:21:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:21:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:21:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:22:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:22:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:22:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:22:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:22:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:22:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:22:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:22:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:22:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:22:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:22:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:22:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:22:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:22:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:22:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:22:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:22:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:22:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:22:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:22:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:22:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:22:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:22:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:22:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:22:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:22:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:22:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:23:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:23:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:23:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:23:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:23:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:23:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:23:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:23:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:23:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:23:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:23:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:23:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:23:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:23:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:23:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:23:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:23:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:23:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:23:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:23:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:23:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:23:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:23:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:23:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:23:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:23:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:23:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:24:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:24:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:24:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:24:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:24:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:24:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:24:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:24:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:24:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:24:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:24:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:24:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:24:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:24:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:24:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:24:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:24:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:24:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:24:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:24:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:24:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:24:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:24:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:24:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:24:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:24:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:24:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:24:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:25:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:25:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:25:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:25:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:25:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:25:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:25:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:25:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:25:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:25:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:25:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:25:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:25:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:25:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:25:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:25:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:25:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:25:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:25:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:25:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:25:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:25:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:25:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:25:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:25:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:25:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:25:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:26:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:26:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:26:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:26:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:26:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:26:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:26:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:26:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:26:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:26:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:26:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:26:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:26:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:26:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:26:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:26:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:26:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:26:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:26:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:26:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:26:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:26:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:26:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:26:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:26:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:26:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:26:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:27:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:27:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:27:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:27:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:27:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:27:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:27:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:27:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:27:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:27:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:27:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:27:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:27:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:27:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:27:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:27:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:27:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:27:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:27:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:27:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:27:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:27:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:27:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:27:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:27:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:27:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:27:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:27:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:28:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:28:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:28:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:28:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:28:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:28:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:28:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:28:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:28:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:28:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:28:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:28:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:28:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:28:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:28:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:28:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:28:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:28:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:28:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:28:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:28:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:28:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:28:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:28:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:28:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:28:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:28:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:29:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:29:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:29:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:29:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:29:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:29:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:29:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:29:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:29:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:29:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:29:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:29:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:29:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:29:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:29:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:29:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:29:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:29:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:29:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:29:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:29:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:29:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:29:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:29:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:29:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:29:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:29:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:30:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:30:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:30:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:30:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:30:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:30:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:30:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:30:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:30:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:30:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:30:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:30:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:30:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:30:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:30:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:30:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:30:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:30:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:30:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:30:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:30:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:30:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:30:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:30:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:30:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:30:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:30:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:30:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:31:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:31:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:31:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:31:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:31:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:31:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:31:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:31:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:31:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:31:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:31:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:31:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:31:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:31:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:31:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:31:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:31:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:31:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:31:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:31:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:31:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:31:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:31:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:31:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:31:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:31:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:31:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:32:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:32:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:32:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:32:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:32:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:32:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:32:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:32:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:32:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:32:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:32:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:32:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:32:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:32:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:32:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:32:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:32:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:32:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:32:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:32:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:32:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:32:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:32:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:32:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:32:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:32:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:32:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:33:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:33:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:33:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:33:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:33:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:33:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:33:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:33:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:33:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:33:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:33:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:33:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:33:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:33:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:33:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:33:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:33:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:33:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:33:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:33:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:33:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:33:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:33:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:33:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:33:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:33:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:33:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:34:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:34:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:34:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:34:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:34:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:34:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:34:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:34:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:34:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:34:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:34:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:34:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:34:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:34:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:34:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:34:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:34:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:34:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:34:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:34:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:34:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:34:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:34:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:34:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:34:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:34:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:34:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:34:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:35:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:35:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:35:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:35:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:35:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:35:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:35:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:35:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:35:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:35:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:35:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:35:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:35:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:35:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:35:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:35:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:35:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:35:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:35:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:35:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:35:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:35:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:35:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:35:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:35:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:35:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:35:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:36:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:36:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:36:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:36:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:36:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:36:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:36:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:36:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:36:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:36:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:36:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:36:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:36:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:36:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:36:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:36:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:36:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:36:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:36:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:36:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:36:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:36:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:36:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:36:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:36:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:36:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:36:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:37:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:37:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:37:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:37:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:37:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:37:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:37:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:37:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:37:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:37:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:37:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:37:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:37:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:37:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:37:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:37:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:37:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:37:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:37:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:37:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:37:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:37:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:37:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:37:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:37:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:37:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:37:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:37:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:38:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:38:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:38:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:38:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:38:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:38:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:38:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:38:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:38:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:38:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:38:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:38:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:38:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:38:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:38:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:38:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:38:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:38:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:38:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:38:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:38:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:38:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:38:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:38:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:38:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:38:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:38:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:39:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:39:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:39:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:39:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:39:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:39:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:39:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:39:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:39:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:39:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:39:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:39:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:39:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:39:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:39:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:39:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:39:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:39:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:39:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:39:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:39:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:39:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:39:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:39:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:39:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:39:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:39:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:40:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:40:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:40:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:40:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:40:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:40:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:40:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:40:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:40:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:40:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:40:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:40:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:40:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:40:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:40:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:40:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:40:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:40:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:40:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:40:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:40:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:40:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:40:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:40:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:40:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:40:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:40:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:40:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:41:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:41:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:41:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:41:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:41:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:41:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:41:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:41:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:41:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:41:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:41:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:41:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:41:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:41:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:41:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:41:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:41:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:41:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:41:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:41:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:41:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:41:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:41:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:41:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:41:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:41:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:41:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:42:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:42:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:42:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:42:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:42:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:42:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:42:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:42:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:42:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:42:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:42:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:42:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:42:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:42:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:42:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:42:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:42:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:42:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:42:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:42:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:42:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:42:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:42:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:42:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:42:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:42:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:42:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:43:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:43:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:43:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:43:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:43:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:43:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:43:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:43:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:43:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:43:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:43:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:43:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:43:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:43:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:43:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:43:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:43:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:43:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:43:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:43:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:43:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:43:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:43:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:43:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:43:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:43:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:43:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:43:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:44:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:44:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:44:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:44:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:44:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:44:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:44:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:44:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:44:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:44:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:44:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:44:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:44:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:44:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:44:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:44:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:44:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:44:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:44:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:44:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:44:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:44:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:44:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:44:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:44:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:44:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:44:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:45:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:45:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:45:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:45:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:45:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:45:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:45:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:45:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:45:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:45:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:45:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:45:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:45:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:45:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:45:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:45:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:45:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:45:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:45:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:45:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:45:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:45:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:45:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:45:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:45:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:45:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:45:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:46:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:46:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:46:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:46:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:46:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:46:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:46:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:46:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:46:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:46:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:46:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:46:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:46:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:46:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:46:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:46:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:46:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:46:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:46:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:46:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:46:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:46:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:46:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:46:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:46:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:46:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:46:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:46:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:47:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:47:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:47:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:47:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:47:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:47:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:47:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:47:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:47:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:47:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:47:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:47:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:47:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:47:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:47:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:47:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:47:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:47:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:47:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:47:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:47:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:47:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:47:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:47:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:47:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:47:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:47:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:48:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:48:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:48:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:48:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:48:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:48:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:48:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:48:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:48:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:48:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:48:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:48:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:48:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:48:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:48:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:48:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:48:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:48:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:48:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:48:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:48:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:48:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:48:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:48:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:48:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:48:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:48:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:49:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:49:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:49:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:49:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:49:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:49:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:49:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:49:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:49:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:49:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:49:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:49:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:49:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:49:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:49:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:49:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:49:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:49:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:49:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:49:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:49:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:49:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:49:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:49:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:49:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:49:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:49:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:50:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:50:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:50:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:50:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:50:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:50:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:50:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:50:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:50:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:50:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:50:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:50:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:50:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:50:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:50:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:50:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:50:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:50:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:50:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:50:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:50:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:50:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:50:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:50:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:50:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:50:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:50:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:50:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:51:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:51:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:51:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:51:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:51:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:51:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:51:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:51:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:51:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:51:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:51:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:51:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:51:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:51:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:51:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:51:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:51:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:51:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:51:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:51:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:51:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:51:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:51:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:51:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:51:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:51:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:51:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:52:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:52:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:52:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:52:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:52:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:52:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:52:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:52:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:52:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:52:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:52:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:52:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:52:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:52:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:52:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:52:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:52:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:52:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:52:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:52:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:52:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:52:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:52:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:52:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:52:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:52:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:52:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:53:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:53:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:53:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:53:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:53:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:53:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:53:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:53:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:53:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:53:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:53:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:53:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:53:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:53:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:53:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:53:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:53:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:53:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:53:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:53:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:53:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:53:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:53:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:53:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:53:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:53:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:53:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:53:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:54:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:54:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:54:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:54:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:54:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:54:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:54:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:54:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:54:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:54:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:54:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:54:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:54:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:54:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:54:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:54:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:54:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:54:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:54:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:54:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:54:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:54:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:54:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:54:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:54:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:54:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:54:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:55:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:55:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:55:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:55:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:55:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:55:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:55:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:55:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:55:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:55:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:55:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:55:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:55:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:55:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:55:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:55:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:55:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:55:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:55:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:55:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:55:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:55:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:55:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:55:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:55:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:55:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:55:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:56:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:56:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:56:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:56:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:56:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:56:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:56:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:56:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:56:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:56:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:56:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:56:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:56:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:56:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:56:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:56:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:56:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:56:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:56:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:56:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:56:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:56:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:56:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:56:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:56:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:56:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:56:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:56:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:57:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:57:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:57:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:57:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:57:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:57:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:57:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:57:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:57:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:57:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:57:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:57:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:57:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:57:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:57:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:57:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:57:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:57:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:57:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:57:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:57:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:57:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:57:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:57:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:57:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:57:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:57:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:58:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:58:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:58:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:58:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:58:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:58:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:58:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:58:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:58:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:58:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:58:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:58:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:58:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:58:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:58:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:58:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:58:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:58:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:58:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:58:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:58:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:58:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:58:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:58:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:58:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:58:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:58:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:59:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:59:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:59:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:59:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:59:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:59:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:59:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:59:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:59:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:59:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:59:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:59:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:59:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:59:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:59:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:59:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:59:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:59:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:59:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 12:59:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:59:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:59:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:59:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:59:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 12:59:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 12:59:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 12:59:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 12:59:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:00:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:00:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:00:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:00:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:00:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:00:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:00:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:00:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:00:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:00:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:00:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:00:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:00:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:00:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:00:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:00:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:00:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:00:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:00:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:00:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:00:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:00:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:00:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:00:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:00:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:00:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:00:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:01:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:01:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:01:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:01:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:01:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:01:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:01:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:01:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:01:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:01:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:01:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:01:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:01:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:01:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:01:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:01:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:01:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:01:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:01:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:01:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:01:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:01:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:01:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:01:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:01:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:01:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:01:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:02:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:02:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:02:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:02:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:02:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:02:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:02:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:02:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:02:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:02:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:02:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:02:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:02:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:02:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:02:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:02:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:02:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:02:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:02:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:02:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:02:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:02:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:02:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:02:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:02:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:02:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:02:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:02:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:03:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:03:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:03:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:03:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:03:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:03:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:03:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:03:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:03:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:03:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:03:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:03:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:03:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:03:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:03:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:03:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:03:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:03:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:03:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:03:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:03:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:03:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:03:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:03:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:03:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:03:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:03:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:04:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:04:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:04:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:04:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:04:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:04:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:04:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:04:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:04:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:04:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:04:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:04:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:04:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:04:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:04:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:04:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:04:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:04:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:04:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:04:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:04:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:04:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:04:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:04:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:04:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:04:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:04:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:05:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:05:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:05:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:05:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:05:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:05:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:05:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:05:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:05:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:05:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:05:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:05:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:05:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:05:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:05:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:05:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:05:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:05:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:05:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:05:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:05:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:05:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:05:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:05:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:05:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:05:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:05:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:06:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:06:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:06:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:06:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:06:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:06:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:06:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:06:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:06:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:06:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:06:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:06:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:06:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:06:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:06:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:06:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:06:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:06:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:06:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:06:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:06:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:06:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:06:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:06:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:06:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:06:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:06:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:06:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:07:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:07:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:07:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:07:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:07:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:07:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:07:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:07:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:07:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:07:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:07:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:07:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:07:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:07:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:07:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:07:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:07:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:07:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:07:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:07:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:07:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:07:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:07:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:07:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:07:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:07:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:07:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:08:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:08:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:08:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:08:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:08:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:08:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:08:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:08:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:08:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:08:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:08:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:08:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:08:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:08:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:08:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:08:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:08:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:08:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:08:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:08:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:08:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:08:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:08:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:08:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:08:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:08:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:08:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:09:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:09:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:09:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:09:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:09:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:09:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:09:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:09:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:09:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:09:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:09:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:09:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:09:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:09:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:09:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:09:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:09:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:09:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:09:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:09:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:09:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:09:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:09:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:09:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:09:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:09:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:09:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:09:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:10:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:10:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:10:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:10:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:10:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:10:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:10:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:10:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:10:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:10:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:10:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:10:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:10:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:10:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:10:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:10:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:10:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:10:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:10:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:10:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:10:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:10:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:10:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:10:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:10:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:10:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:10:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:11:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:11:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:11:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:11:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:11:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:11:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:11:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:11:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:11:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:11:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:11:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:11:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:11:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:11:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:11:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:11:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:11:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:11:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:11:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:11:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:11:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:11:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:11:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:11:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:11:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:11:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:11:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:12:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:12:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:12:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:12:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:12:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:12:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:12:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:12:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:12:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:12:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:12:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:12:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:12:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:12:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:12:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:12:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:12:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:12:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:12:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:12:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:12:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:12:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:12:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:12:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:12:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:12:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:12:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:12:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:13:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:13:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:13:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:13:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:13:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:13:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:13:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:13:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:13:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:13:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:13:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:13:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:13:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:13:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:13:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:13:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:13:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:13:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:13:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:13:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:13:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:13:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:13:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:13:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:13:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:13:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:13:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:14:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:14:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:14:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:14:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:14:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:14:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:14:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:14:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:14:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:14:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:14:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:14:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:14:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:14:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:14:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:14:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:14:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:14:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:14:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:14:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:14:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:14:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:14:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:14:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:14:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:14:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:14:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:15:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:15:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:15:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:15:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:15:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:15:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:15:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:15:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:15:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:15:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:15:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:15:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:15:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:15:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:15:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:15:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:15:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:15:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:15:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:15:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:15:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:15:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:15:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:15:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:15:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:15:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:15:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:15:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:16:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:16:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:16:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:16:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:16:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:16:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:16:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:16:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:16:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:16:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:16:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:16:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:16:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:16:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:16:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:16:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:16:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:16:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:16:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:16:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:16:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:16:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:16:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:16:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:16:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:16:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:16:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:17:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:17:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:17:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:17:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:17:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:17:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:17:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:17:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:17:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:17:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:17:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:17:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:17:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:17:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:17:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:17:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:17:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:17:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:17:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:17:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:17:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:17:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:17:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:17:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:17:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:17:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:17:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:18:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:18:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:18:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:18:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:18:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:18:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:18:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:18:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:18:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:18:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:18:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:18:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:18:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:18:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:18:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:18:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:18:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:18:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:18:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:18:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:18:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:18:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:18:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:18:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:18:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:18:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:18:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:18:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:19:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:19:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:19:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:19:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:19:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:19:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:19:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:19:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:19:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:19:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:19:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:19:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:19:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:19:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:19:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:19:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:19:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:19:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:19:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:19:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:19:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:19:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:19:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:19:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:19:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:19:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:19:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:20:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:20:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:20:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:20:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:20:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:20:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:20:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:20:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:20:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:20:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:20:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:20:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:20:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:20:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:20:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:20:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:20:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:20:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:20:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:20:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:20:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:20:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:20:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:20:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:20:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:20:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:20:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:21:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:21:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:21:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:21:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:21:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:21:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:21:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:21:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:21:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:21:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:21:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:21:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:21:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:21:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:21:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:21:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:21:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:21:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:21:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:21:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:21:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:21:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:21:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:21:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:21:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:21:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:21:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:22:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:22:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:22:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:22:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:22:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:22:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:22:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:22:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:22:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:22:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:22:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:22:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:22:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:22:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:22:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:22:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:22:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:22:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:22:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:22:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:22:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:22:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:22:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:22:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:22:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:22:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:22:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:22:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:23:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:23:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:23:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:23:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:23:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:23:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:23:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:23:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:23:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:23:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:23:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:23:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:23:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:23:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:23:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:23:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:23:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:23:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:23:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:23:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:23:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:23:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:23:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:23:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:23:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:23:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:23:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:24:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:24:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:24:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:24:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:24:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:24:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:24:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:24:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:24:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:24:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:24:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:24:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:24:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:24:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:24:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:24:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:24:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:24:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:24:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:24:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:24:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:24:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:24:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:24:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:24:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:24:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:24:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:25:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:25:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:25:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:25:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:25:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:25:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:25:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:25:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:25:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:25:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:25:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:25:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:25:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:25:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:25:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:25:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:25:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:25:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:25:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:25:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:25:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:25:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:25:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:25:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:25:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:25:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:25:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:25:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:26:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:26:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:26:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:26:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:26:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:26:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:26:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:26:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:26:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:26:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:26:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:26:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:26:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:26:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:26:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:26:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:26:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:26:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:26:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:26:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:26:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:26:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:26:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:26:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:26:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:26:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:26:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:27:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:27:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:27:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:27:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:27:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:27:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:27:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:27:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:27:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:27:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:27:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:27:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:27:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:27:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:27:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:27:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:27:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:27:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:27:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:27:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:27:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:27:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:27:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:27:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:27:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:27:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:27:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:28:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:28:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:28:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:28:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:28:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:28:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:28:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:28:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:28:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:28:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:28:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:28:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:28:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:28:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:28:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:28:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:28:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:28:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:28:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:28:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:28:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:28:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:28:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:28:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:28:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:28:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:28:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:28:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:29:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:29:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:29:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:29:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:29:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:29:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:29:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:29:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:29:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:29:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:29:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:29:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:29:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:29:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:29:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:29:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:29:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:29:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:29:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:29:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:29:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:29:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:29:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:29:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:29:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:29:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:29:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:30:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:30:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:30:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:30:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:30:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:30:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:30:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:30:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:30:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:30:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:30:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:30:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:30:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:30:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:30:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:30:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:30:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:30:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:30:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:30:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:30:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:30:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:30:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:30:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:30:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:30:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:30:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:31:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:31:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:31:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:31:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:31:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:31:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:31:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:31:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:31:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:31:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:31:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:31:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:31:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:31:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:31:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:31:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:31:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:31:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:31:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:31:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:31:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:31:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:31:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:31:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:31:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:31:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:31:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:31:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:32:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:32:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:32:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:32:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:32:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:32:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:32:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:32:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:32:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:32:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:32:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:32:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:32:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:32:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:32:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:32:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:32:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:32:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:32:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:32:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:32:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:32:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:32:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:32:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:32:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:32:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:32:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:33:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:33:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:33:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:33:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:33:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:33:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:33:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:33:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:33:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:33:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:33:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:33:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:33:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:33:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:33:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:33:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:33:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:33:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:33:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:33:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:33:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:33:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:33:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:33:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:33:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:33:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:33:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:34:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:34:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:34:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:34:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:34:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:34:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:34:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:34:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:34:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:34:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:34:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:34:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:34:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:34:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:34:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:34:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:34:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:34:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:34:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:34:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:34:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:34:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:34:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:34:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:34:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:34:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:34:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:34:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:35:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:35:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:35:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:35:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:35:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:35:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:35:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:35:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:35:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:35:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:35:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:35:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:35:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:35:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:35:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:35:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:35:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:35:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:35:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:35:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:35:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:35:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:35:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:35:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:35:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:35:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:35:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:36:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:36:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:36:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:36:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:36:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:36:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:36:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:36:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:36:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:36:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:36:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:36:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:36:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:36:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:36:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:36:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:36:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:36:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:36:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:36:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:36:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:36:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:36:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:36:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:36:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:36:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:36:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:37:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:37:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:37:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:37:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:37:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:37:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:37:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:37:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:37:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:37:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:37:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:37:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:37:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:37:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:37:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:37:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:37:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:37:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:37:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:37:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:37:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:37:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:37:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:37:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:37:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:37:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:37:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:37:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:38:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:38:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:38:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:38:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:38:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:38:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:38:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:38:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:38:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:38:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:38:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:38:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:38:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:38:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:38:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:38:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:38:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:38:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:38:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:38:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:38:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:38:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:38:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:38:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:38:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:38:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:38:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:39:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:39:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:39:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:39:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:39:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:39:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:39:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:39:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:39:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:39:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:39:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:39:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:39:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:39:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:39:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:39:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:39:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:39:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:39:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:39:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:39:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:39:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:39:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:39:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:39:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:39:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:39:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:40:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:40:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:40:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:40:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:40:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:40:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:40:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:40:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:40:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:40:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:40:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:40:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:40:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:40:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:40:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:40:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:40:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:40:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:40:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:40:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:40:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:40:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:40:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:40:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:40:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:40:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:40:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:41:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:41:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:41:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:41:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:41:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:41:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:41:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:41:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:41:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:41:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:41:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:41:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:41:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:41:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:41:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:41:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:41:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:41:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:41:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:41:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:41:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:41:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:41:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:41:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:41:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:41:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:41:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:41:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:42:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:42:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:42:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:42:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:42:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:42:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:42:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:42:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:42:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:42:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:42:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:42:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:42:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:42:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:42:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:42:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:42:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:42:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:42:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:42:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:42:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:42:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:42:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:42:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:42:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:42:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:42:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:43:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:43:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:43:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:43:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:43:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:43:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:43:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:43:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:43:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:43:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:43:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:43:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:43:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:43:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:43:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:43:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:43:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:43:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:43:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:43:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:43:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:43:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:43:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:43:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:43:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:43:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:43:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:44:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:44:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:44:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:44:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:44:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:44:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:44:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:44:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:44:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:44:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:44:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:44:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:44:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:44:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:44:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:44:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:44:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:44:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:44:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:44:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:44:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:44:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:44:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:44:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:44:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:44:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:44:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:44:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:45:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:45:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:45:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:45:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:45:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:45:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:45:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:45:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:45:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:45:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:45:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:45:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:45:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:45:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:45:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:45:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:45:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:45:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:45:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:45:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:45:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:45:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:45:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:45:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:45:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:45:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:45:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:46:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:46:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:46:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:46:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:46:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:46:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:46:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:46:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:46:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:46:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:46:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:46:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:46:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:46:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:46:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:46:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:46:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:46:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:46:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:46:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:46:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:46:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:46:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:46:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:46:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:46:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:46:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:47:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:47:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:47:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:47:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:47:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:47:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:47:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:47:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:47:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:47:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:47:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:47:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:47:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:47:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:47:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:47:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:47:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:47:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:47:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:47:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:47:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:47:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:47:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:47:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:47:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:47:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:47:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:47:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:48:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:48:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:48:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:48:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:48:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:48:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:48:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:48:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:48:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:48:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:48:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:48:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:48:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:48:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:48:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:48:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:48:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:48:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:48:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:48:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:48:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:48:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:48:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:48:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:48:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:48:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:48:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:49:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:49:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:49:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:49:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:49:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:49:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:49:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:49:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:49:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:49:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:49:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:49:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:49:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:49:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:49:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:49:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:49:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:49:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:49:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:49:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:49:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:49:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:49:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:49:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:49:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:49:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:49:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:50:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  7 13:50:01 rdma06 dnf[581751]: Updating Subscription Management repositories.
Apr  7 13:50:01 rdma06 dnf[581751]: Unable to read consumer identity
Apr  7 13:50:01 rdma06 dnf[581751]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  7 13:50:02 rdma06 dnf[581751]: Copr repo for fmf owned by psss                 3.6 kB/s | 3.3 kB     00:00
Apr  7 13:50:02 rdma06 dnf[581751]: beaker-AppStream                                 35 kB/s | 3.2 kB     00:00
Apr  7 13:50:02 rdma06 dnf[581751]: beaker-AppStream-debuginfo                       34 kB/s | 1.5 kB     00:00
Apr  7 13:50:02 rdma06 dnf[581751]: beaker-BaseOS                                   110 kB/s | 2.7 kB     00:00
Apr  7 13:50:02 rdma06 dnf[581751]: beaker-BaseOS-debuginfo                          50 kB/s | 1.5 kB     00:00
Apr  7 13:50:02 rdma06 dnf[581751]: beaker-CRB                                       23 kB/s | 2.7 kB     00:00
Apr  7 13:50:03 rdma06 dnf[581751]: beaker-CRB-debuginfo                             36 kB/s | 1.5 kB     00:00
Apr  7 13:50:03 rdma06 dnf[581751]: beaker-HighAvailability                          57 kB/s | 2.8 kB     00:00
Apr  7 13:50:03 rdma06 dnf[581751]: beaker-HighAvailability-debuginfo                37 kB/s | 1.5 kB     00:00
Apr  7 13:50:03 rdma06 dnf[581751]: beaker-NFV                                       67 kB/s | 2.7 kB     00:00
Apr  7 13:50:03 rdma06 dnf[581751]: beaker-NFV-debuginfo                             45 kB/s | 1.5 kB     00:00
Apr  7 13:50:03 rdma06 dnf[581751]: beaker-RT                                        79 kB/s | 2.7 kB     00:00
Apr  7 13:50:03 rdma06 dnf[581751]: beaker-RT-debuginfo                              63 kB/s | 1.5 kB     00:00
Apr  7 13:50:03 rdma06 dnf[581751]: beaker-ResilientStorage                          73 kB/s | 2.8 kB     00:00
Apr  7 13:50:03 rdma06 dnf[581751]: beaker-ResilientStorage-debuginfo                69 kB/s | 1.5 kB     00:00
Apr  7 13:50:03 rdma06 dnf[581751]: beaker-SAP                                      136 kB/s | 2.7 kB     00:00
Apr  7 13:50:03 rdma06 dnf[581751]: beaker-SAP-debuginfo                             59 kB/s | 1.5 kB     00:00
Apr  7 13:50:03 rdma06 dnf[581751]: beaker-SAPHANA                                   48 kB/s | 2.7 kB     00:00
Apr  7 13:50:03 rdma06 dnf[581751]: beaker-SAPHANA-debuginfo                         40 kB/s | 1.5 kB     00:00
Apr  7 13:50:04 rdma06 dnf[581751]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr  7 13:50:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:50:04 rdma06 dnf[581751]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr  7 13:50:05 rdma06 dnf[581751]: Metadata cache created.
Apr  7 13:50:05 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  7 13:50:05 rdma06 systemd[1]: Finished dnf makecache.
Apr  7 13:50:05 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.407s CPU time.
Apr  7 13:50:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:50:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:50:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:50:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:50:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:50:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:50:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:50:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:50:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:50:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:50:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:50:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:50:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:50:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:50:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:50:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:50:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:50:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:50:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:50:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:50:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:50:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:50:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:50:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:50:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:50:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:50:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:51:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:51:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:51:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:51:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:51:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:51:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:51:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:51:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:51:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:51:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:51:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:51:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:51:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:51:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:51:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:51:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:51:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:51:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:51:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:51:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:51:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:51:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:51:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:51:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:51:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:51:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:51:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:52:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:52:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:52:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:52:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:52:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:52:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:52:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:52:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:52:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:52:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:52:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:52:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:52:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:52:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:52:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:52:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:52:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:52:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:52:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:52:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:52:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:52:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:52:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:52:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:52:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:52:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:52:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:53:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:53:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:53:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:53:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:53:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:53:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:53:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:53:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:53:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:53:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:53:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:53:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:53:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:53:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:53:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:53:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:53:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:53:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:53:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:53:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:53:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:53:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:53:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:53:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:53:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:53:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:53:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:53:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:54:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:54:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:54:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:54:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:54:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:54:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:54:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:54:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:54:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:54:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:54:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:54:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:54:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:54:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:54:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:54:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:54:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:54:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:54:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:54:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:54:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:54:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:54:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:54:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:54:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:54:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:54:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:55:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:55:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:55:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:55:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:55:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:55:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:55:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:55:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:55:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:55:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:55:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:55:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:55:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:55:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:55:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:55:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:55:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:55:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:55:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:55:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:55:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:55:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:55:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:55:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:55:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:55:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:55:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:56:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:56:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:56:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:56:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:56:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:56:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:56:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:56:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:56:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:56:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:56:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:56:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:56:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:56:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:56:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:56:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:56:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:56:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:56:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:56:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:56:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:56:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:56:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:56:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:56:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:56:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:56:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:57:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:57:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:57:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:57:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:57:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:57:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:57:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:57:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:57:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:57:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:57:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:57:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:57:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:57:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:57:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:57:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:57:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:57:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:57:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:57:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:57:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:57:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:57:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:57:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:57:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:57:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:57:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:57:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:58:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:58:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:58:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:58:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:58:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:58:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:58:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:58:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:58:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:58:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:58:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:58:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:58:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:58:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:58:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:58:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:58:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:58:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:58:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:58:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:58:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:58:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:58:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:58:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:58:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:58:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:58:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:59:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:59:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:59:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:59:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:59:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:59:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:59:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:59:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:59:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:59:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:59:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:59:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:59:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:59:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:59:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:59:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:59:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:59:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:59:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 13:59:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:59:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:59:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:59:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 13:59:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 13:59:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 13:59:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 13:59:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:00:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:00:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:00:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:00:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:00:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:00:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:00:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:00:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:00:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:00:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:00:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:00:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:00:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:00:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:00:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:00:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:00:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:00:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:00:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:00:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:00:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:00:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:00:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:00:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:00:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:00:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:00:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:00:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:01:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:01:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:01:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:01:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:01:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:01:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:01:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:01:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:01:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:01:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:01:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:01:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:01:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:01:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:01:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:01:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:01:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:01:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:01:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:01:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:01:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:01:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:01:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:01:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:01:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:01:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:01:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:02:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:02:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:02:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:02:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:02:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:02:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:02:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:02:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:02:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:02:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:02:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:02:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:02:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:02:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:02:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:02:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:02:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:02:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:02:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:02:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:02:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:02:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:02:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:02:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:02:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:02:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:02:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:03:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:03:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:03:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:03:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:03:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:03:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:03:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:03:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:03:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:03:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:03:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:03:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:03:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:03:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:03:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:03:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:03:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:03:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:03:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:03:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:03:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:03:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:03:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:03:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:03:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:03:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:03:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:03:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:04:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:04:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:04:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:04:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:04:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:04:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:04:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:04:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:04:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:04:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:04:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:04:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:04:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:04:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:04:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:04:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:04:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:04:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:04:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:04:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:04:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:04:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:04:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:04:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:04:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:04:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:04:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:05:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:05:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:05:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:05:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:05:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:05:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:05:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:05:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:05:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:05:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:05:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:05:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:05:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:05:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:05:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:05:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:05:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:05:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:05:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:05:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:05:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:05:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:05:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:05:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:05:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:05:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:05:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:06:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:06:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:06:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:06:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:06:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:06:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:06:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:06:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:06:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:06:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:06:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:06:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:06:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:06:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:06:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:06:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:06:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:06:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:06:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:06:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:06:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:06:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:06:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:06:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:06:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:06:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:06:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:06:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:07:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:07:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:07:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:07:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:07:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:07:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:07:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:07:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:07:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:07:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:07:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:07:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:07:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:07:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:07:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:07:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:07:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:07:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:07:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:07:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:07:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:07:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:07:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:07:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:07:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:07:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:07:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:08:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:08:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:08:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:08:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:08:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:08:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:08:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:08:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:08:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:08:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:08:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:08:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:08:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:08:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:08:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:08:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:08:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:08:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:08:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:08:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:08:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:08:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:08:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:08:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:08:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:08:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:08:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:09:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:09:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:09:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:09:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:09:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:09:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:09:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:09:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:09:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:09:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:09:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:09:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:09:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:09:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:09:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:09:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:09:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:09:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:09:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:09:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:09:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:09:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:09:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:09:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:09:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:09:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:09:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:09:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:10:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:10:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:10:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:10:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:10:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:10:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:10:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:10:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:10:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:10:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:10:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:10:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:10:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:10:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:10:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:10:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:10:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:10:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:10:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:10:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:10:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:10:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:10:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:10:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:10:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:10:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:10:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:11:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:11:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:11:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:11:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:11:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:11:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:11:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:11:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:11:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:11:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:11:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:11:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:11:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:11:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:11:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:11:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:11:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:11:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:11:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:11:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:11:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:11:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:11:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:11:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:11:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:11:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:11:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:12:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:12:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:12:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:12:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:12:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:12:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:12:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:12:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:12:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:12:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:12:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:12:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:12:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:12:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:12:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:12:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:12:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:12:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:12:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:12:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:12:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:12:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:12:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:12:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:12:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:12:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:12:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:13:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:13:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:13:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:13:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:13:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:13:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:13:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:13:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:13:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:13:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:13:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:13:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:13:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:13:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:13:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:13:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:13:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:13:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:13:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:13:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:13:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:13:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:13:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:13:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:13:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:13:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:13:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:13:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:14:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:14:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:14:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:14:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:14:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:14:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:14:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:14:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:14:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:14:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:14:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:14:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:14:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:14:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:14:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:14:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:14:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:14:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:14:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:14:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:14:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:14:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:14:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:14:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:14:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:14:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:14:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:15:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:15:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:15:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:15:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:15:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:15:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:15:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:15:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:15:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:15:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:15:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:15:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:15:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:15:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:15:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:15:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:15:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:15:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:15:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:15:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:15:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:15:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:15:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:15:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:15:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:15:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:15:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:16:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:16:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:16:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:16:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:16:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:16:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:16:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:16:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:16:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:16:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:16:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:16:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:16:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:16:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:16:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:16:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:16:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:16:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:16:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:16:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:16:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:16:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:16:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:16:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:16:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:16:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:16:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:16:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:17:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:17:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:17:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:17:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:17:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:17:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:17:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:17:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:17:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:17:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:17:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:17:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:17:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:17:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:17:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:17:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:17:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:17:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:17:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:17:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:17:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:17:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:17:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:17:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:17:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:17:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:17:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:18:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:18:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:18:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:18:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:18:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:18:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:18:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:18:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:18:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:18:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:18:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:18:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:18:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:18:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:18:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:18:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:18:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:18:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:18:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:18:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:18:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:18:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:18:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:18:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:18:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:18:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:18:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:19:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:19:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:19:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:19:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:19:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:19:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:19:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:19:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:19:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:19:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:19:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:19:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:19:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:19:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:19:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:19:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:19:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:19:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:19:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:19:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:19:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:19:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:19:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:19:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:19:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:19:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:19:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:19:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:20:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:20:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:20:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:20:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:20:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:20:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:20:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:20:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:20:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:20:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:20:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:20:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:20:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:20:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:20:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:20:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:20:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:20:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:20:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:20:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:20:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:20:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:20:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:20:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:20:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:20:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:20:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:21:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:21:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:21:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:21:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:21:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:21:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:21:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:21:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:21:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:21:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:21:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:21:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:21:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:21:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:21:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:21:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:21:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:21:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:21:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:21:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:21:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:21:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:21:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:21:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:21:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:21:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:21:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:22:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:22:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:22:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:22:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:22:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:22:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:22:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:22:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:22:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:22:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:22:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:22:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:22:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:22:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:22:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:22:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:22:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:22:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:22:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:22:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:22:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:22:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:22:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:22:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:22:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:22:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:22:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:22:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:23:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:23:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:23:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:23:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:23:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:23:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:23:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:23:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:23:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:23:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:23:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:23:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:23:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:23:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:23:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:23:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:23:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:23:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:23:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:23:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:23:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:23:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:23:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:23:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:23:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:23:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:23:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:24:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:24:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:24:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:24:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:24:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:24:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:24:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:24:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:24:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:24:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:24:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:24:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:24:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:24:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:24:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:24:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:24:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:24:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:24:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:24:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:24:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:24:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:24:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:24:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:24:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:24:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:24:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:25:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:25:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:25:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:25:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:25:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:25:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:25:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:25:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:25:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:25:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:25:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:25:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:25:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:25:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:25:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:25:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:25:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:25:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:25:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:25:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:25:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:25:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:25:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:25:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:25:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:25:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:25:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:25:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:26:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:26:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:26:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:26:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:26:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:26:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:26:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:26:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:26:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:26:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:26:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:26:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:26:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:26:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:26:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:26:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:26:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:26:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:26:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:26:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:26:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:26:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:26:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:26:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:26:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:26:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:26:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:27:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:27:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:27:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:27:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:27:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:27:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:27:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:27:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:27:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:27:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:27:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:27:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:27:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:27:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:27:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:27:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:27:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:27:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:27:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:27:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:27:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:27:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:27:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:27:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:27:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:27:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:27:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:28:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:28:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:28:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:28:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:28:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:28:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:28:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:28:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:28:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:28:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:28:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:28:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:28:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:28:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:28:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:28:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:28:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:28:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:28:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:28:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:28:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:28:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:28:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:28:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:28:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:28:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:28:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:29:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:29:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:29:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:29:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:29:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:29:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:29:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:29:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:29:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:29:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:29:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:29:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:29:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:29:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:29:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:29:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:29:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:29:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:29:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:29:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:29:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:29:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:29:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:29:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:29:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:29:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:29:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:29:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:30:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:30:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:30:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:30:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:30:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:30:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:30:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:30:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:30:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:30:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:30:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:30:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:30:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:30:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:30:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:30:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:30:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:30:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:30:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:30:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:30:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:30:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:30:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:30:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:30:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:30:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:30:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:31:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:31:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:31:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:31:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:31:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:31:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:31:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:31:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:31:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:31:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:31:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:31:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:31:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:31:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:31:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:31:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:31:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:31:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:31:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:31:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:31:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:31:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:31:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:31:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:31:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:31:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:31:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:32:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:32:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:32:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:32:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:32:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:32:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:32:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:32:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:32:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:32:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:32:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:32:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:32:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:32:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:32:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:32:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:32:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:32:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:32:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:32:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:32:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:32:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:32:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:32:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:32:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:32:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:32:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:32:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:33:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:33:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:33:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:33:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:33:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:33:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:33:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:33:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:33:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:33:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:33:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:33:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:33:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:33:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:33:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:33:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:33:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:33:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:33:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:33:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:33:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:33:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:33:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:33:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:33:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:33:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:33:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:34:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:34:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:34:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:34:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:34:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:34:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:34:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:34:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:34:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:34:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:34:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:34:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:34:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:34:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:34:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:34:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:34:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:34:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:34:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:34:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:34:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:34:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:34:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:34:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:34:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:34:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:34:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:35:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:35:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:35:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:35:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:35:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:35:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:35:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:35:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:35:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:35:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:35:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:35:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:35:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:35:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:35:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:35:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:35:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:35:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:35:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:35:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:35:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:35:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:35:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:35:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:35:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:35:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:35:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:35:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:36:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:36:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:36:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:36:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:36:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:36:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:36:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:36:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:36:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:36:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:36:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:36:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:36:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:36:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:36:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:36:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:36:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:36:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:36:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:36:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:36:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:36:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:36:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:36:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:36:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:36:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:36:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:37:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:37:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:37:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:37:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:37:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:37:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:37:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:37:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:37:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:37:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:37:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:37:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:37:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:37:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:37:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:37:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:37:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:37:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:37:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:37:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:37:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:37:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:37:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:37:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:37:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:37:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:37:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:38:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:38:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:38:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:38:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:38:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:38:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:38:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:38:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:38:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:38:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:38:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:38:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:38:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:38:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:38:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:38:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:38:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:38:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:38:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:38:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:38:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:38:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:38:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:38:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:38:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:38:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:38:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:38:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:39:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:39:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:39:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:39:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:39:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:39:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:39:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:39:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:39:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:39:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:39:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:39:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:39:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:39:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:39:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:39:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:39:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:39:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:39:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:39:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:39:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:39:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:39:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:39:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:39:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:39:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:39:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:40:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:40:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:40:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:40:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:40:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:40:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:40:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:40:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:40:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:40:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:40:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:40:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:40:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:40:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:40:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:40:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:40:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:40:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:40:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:40:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:40:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:40:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:40:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:40:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:40:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:40:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:40:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:41:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:41:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:41:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:41:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:41:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:41:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:41:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:41:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:41:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:41:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:41:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:41:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:41:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:41:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:41:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:41:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:41:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:41:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:41:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:41:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:41:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:41:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:41:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:41:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:41:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:41:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:41:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:41:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:42:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:42:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:42:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:42:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:42:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:42:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:42:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:42:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:42:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:42:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:42:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:42:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:42:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:42:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:42:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:42:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:42:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:42:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:42:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:42:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:42:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:42:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:42:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:42:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:42:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:42:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:42:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:43:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:43:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:43:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:43:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:43:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:43:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:43:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:43:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:43:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:43:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:43:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:43:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:43:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:43:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:43:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:43:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:43:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:43:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:43:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:43:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:43:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:43:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:43:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:43:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:43:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:43:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:43:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:44:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:44:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:44:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:44:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:44:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:44:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:44:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:44:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:44:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:44:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:44:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:44:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:44:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:44:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:44:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:44:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:44:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:44:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:44:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:44:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:44:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:44:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:44:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:44:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:44:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:44:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:44:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:44:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:45:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:45:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:45:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:45:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:45:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:45:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:45:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:45:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:45:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:45:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:45:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:45:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:45:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:45:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:45:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:45:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:45:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:45:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:45:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:45:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:45:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:45:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:45:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:45:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:45:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:45:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:45:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:46:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:46:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:46:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:46:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:46:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:46:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:46:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:46:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:46:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:46:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:46:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:46:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:46:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:46:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:46:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:46:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:46:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:46:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:46:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:46:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:46:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:46:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:46:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:46:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:46:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:46:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:46:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:47:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:47:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:47:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:47:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:47:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:47:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:47:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:47:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:47:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:47:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:47:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:47:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:47:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:47:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:47:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:47:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:47:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:47:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:47:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:47:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:47:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:47:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:47:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:47:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:47:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:47:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:47:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:48:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:48:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:48:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:48:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:48:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:48:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:48:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:48:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:48:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:48:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:48:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:48:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:48:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:48:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:48:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:48:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:48:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:48:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:48:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:48:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:48:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:48:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:48:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:48:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:48:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:48:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:48:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:48:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:49:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:49:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:49:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:49:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:49:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:49:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:49:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:49:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:49:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:49:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:49:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:49:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:49:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:49:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:49:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:49:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:49:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:49:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:49:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:49:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:49:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:49:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:49:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:49:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:49:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:49:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:49:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:50:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:50:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:50:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:50:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:50:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:50:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:50:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:50:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:50:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:50:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:50:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:50:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:50:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:50:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:50:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:50:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:50:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:50:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:50:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:50:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:50:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:50:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:50:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:50:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:50:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:50:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:50:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:51:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:51:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:51:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:51:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:51:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:51:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:51:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:51:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:51:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:51:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:51:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:51:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:51:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:51:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:51:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:51:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:51:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:51:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:51:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:51:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:51:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:51:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:51:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:51:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:51:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:51:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:51:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:51:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:52:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:52:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:52:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:52:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:52:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:52:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:52:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:52:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:52:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:52:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:52:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:52:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:52:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:52:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:52:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:52:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:52:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:52:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:52:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:52:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:52:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:52:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:52:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:52:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:52:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:52:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:52:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:53:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:53:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:53:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:53:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:53:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:53:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:53:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:53:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:53:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:53:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:53:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:53:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:53:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:53:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:53:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:53:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:53:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:53:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:53:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:53:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:53:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:53:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:53:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:53:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:53:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:53:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:53:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:54:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:54:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:54:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:54:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:54:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:54:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:54:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:54:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:54:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:54:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:54:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:54:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:54:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:54:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:54:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:54:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:54:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:54:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:54:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:54:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:54:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:54:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:54:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:54:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:54:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:54:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:54:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:54:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:55:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:55:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:55:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:55:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:55:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:55:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:55:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:55:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:55:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:55:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:55:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:55:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:55:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:55:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:55:35 rdma06 chronyd[1034]: Selected source 10.13.199.1
Apr  7 14:55:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:55:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:55:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:55:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:55:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:55:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:55:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:55:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:55:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:55:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:55:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:55:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:55:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:56:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:56:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:56:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:56:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:56:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:56:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:56:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:56:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:56:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:56:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:56:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:56:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:56:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:56:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:56:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:56:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:56:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:56:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:56:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:56:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:56:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:56:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:56:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:56:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:56:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:56:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:56:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:57:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:57:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:57:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:57:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:57:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:57:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:57:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:57:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:57:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:57:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:57:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:57:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:57:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:57:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:57:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:57:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:57:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:57:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:57:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:57:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:57:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:57:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:57:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:57:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:57:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:57:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:57:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:57:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:58:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:58:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:58:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:58:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:58:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:58:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:58:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:58:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:58:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:58:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:58:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:58:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:58:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:58:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:58:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:58:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:58:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:58:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:58:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:58:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:58:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:58:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:58:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:58:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:58:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:58:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:58:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:59:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:59:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:59:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:59:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:59:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:59:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:59:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:59:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:59:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:59:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:59:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:59:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:59:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:59:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:59:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:59:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:59:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:59:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:59:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:59:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:59:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:59:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 14:59:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 14:59:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 14:59:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 14:59:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 14:59:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:00:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:00:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:00:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:00:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:00:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:00:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:00:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:00:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:00:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:00:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:00:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:00:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:00:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:00:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:00:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:00:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:00:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:00:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:00:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:00:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:00:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:00:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:00:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:00:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:00:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:00:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:00:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:00:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:01:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:01:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:01:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:01:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:01:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:01:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:01:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:01:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:01:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:01:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:01:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:01:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:01:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:01:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:01:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:01:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:01:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:01:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:01:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:01:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:01:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:01:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:01:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:01:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:01:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:01:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:01:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:02:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:02:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:02:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:02:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:02:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:02:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:02:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:02:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:02:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:02:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:02:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:02:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:02:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:02:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:02:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:02:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:02:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:02:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:02:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:02:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:02:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:02:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:02:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:02:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:02:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:02:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:02:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:03:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:03:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:03:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:03:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:03:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:03:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:03:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:03:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:03:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:03:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:03:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:03:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:03:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:03:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:03:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:03:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:03:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:03:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:03:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:03:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:03:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:03:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:03:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:03:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:03:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:03:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:03:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:04:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:04:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:04:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:04:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:04:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:04:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:04:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:04:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:04:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:04:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:04:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:04:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:04:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:04:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:04:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:04:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:04:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:04:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:04:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:04:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:04:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:04:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:04:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:04:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:04:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:04:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:04:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:04:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:05:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:05:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:05:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:05:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:05:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:05:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:05:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:05:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:05:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:05:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:05:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:05:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:05:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:05:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:05:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:05:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:05:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:05:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:05:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:05:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:05:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:05:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:05:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:05:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:05:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:05:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:05:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:06:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:06:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:06:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:06:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:06:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:06:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:06:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:06:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:06:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:06:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:06:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:06:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:06:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:06:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:06:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:06:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:06:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:06:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:06:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:06:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:06:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:06:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:06:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:06:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:06:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:06:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:06:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:07:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:07:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:07:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:07:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:07:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:07:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:07:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:07:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:07:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:07:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:07:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:07:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:07:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:07:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:07:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:07:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:07:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:07:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:07:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:07:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:07:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:07:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:07:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:07:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:07:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:07:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:07:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:07:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:08:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:08:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:08:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:08:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:08:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:08:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:08:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:08:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:08:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:08:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:08:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:08:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:08:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:08:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:08:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:08:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:08:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:08:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:08:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:08:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:08:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:08:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:08:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:08:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:08:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:08:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:08:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:09:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:09:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:09:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:09:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:09:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:09:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:09:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:09:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:09:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:09:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:09:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:09:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:09:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:09:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:09:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:09:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:09:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:09:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:09:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:09:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:09:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:09:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:09:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:09:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:09:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:09:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:09:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:10:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:10:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:10:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:10:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:10:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:10:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:10:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:10:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:10:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:10:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:10:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:10:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:10:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:10:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:10:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:10:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:10:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:10:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:10:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:10:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:10:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:10:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:10:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:10:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:10:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:10:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:10:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:10:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:11:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:11:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:11:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:11:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:11:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:11:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:11:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:11:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:11:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:11:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:11:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:11:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:11:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:11:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:11:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:11:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:11:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:11:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:11:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:11:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:11:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:11:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:11:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:11:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:11:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:11:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:11:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:12:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:12:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:12:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:12:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:12:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:12:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:12:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:12:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:12:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:12:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:12:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:12:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:12:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:12:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:12:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:12:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:12:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:12:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:12:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:12:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:12:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:12:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:12:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:12:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:12:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:12:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:12:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:13:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:13:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:13:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:13:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:13:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:13:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:13:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:13:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:13:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:13:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:13:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:13:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:13:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:13:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:13:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:13:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:13:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:13:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:13:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:13:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:13:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:13:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:13:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:13:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:13:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:13:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:13:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:13:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:14:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:14:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:14:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:14:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:14:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:14:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:14:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:14:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:14:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:14:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:14:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:14:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:14:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:14:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:14:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:14:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:14:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:14:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:14:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:14:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:14:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:14:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:14:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:14:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:14:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:14:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:14:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:15:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:15:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:15:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:15:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:15:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:15:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:15:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:15:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:15:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:15:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:15:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:15:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:15:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:15:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:15:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:15:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:15:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:15:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:15:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:15:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:15:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:15:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:15:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:15:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:15:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:15:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:15:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:16:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:16:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:16:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:16:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:16:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:16:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:16:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:16:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:16:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:16:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:16:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:16:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:16:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:16:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:16:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:16:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:16:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:16:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:16:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:16:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:16:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:16:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:16:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:16:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:16:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:16:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:16:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:16:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:17:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:17:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:17:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:17:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:17:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:17:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:17:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:17:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:17:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:17:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:17:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:17:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:17:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:17:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:17:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:17:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:17:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:17:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:17:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:17:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:17:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:17:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:17:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:17:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:17:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:17:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:17:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:18:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:18:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:18:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:18:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:18:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:18:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:18:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:18:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:18:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:18:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:18:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:18:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:18:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:18:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:18:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:18:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:18:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:18:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:18:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:18:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:18:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:18:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:18:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:18:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:18:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:18:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:18:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:19:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:19:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:19:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:19:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:19:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:19:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:19:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:19:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:19:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:19:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:19:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:19:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:19:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:19:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:19:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:19:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:19:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:19:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:19:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:19:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:19:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:19:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:19:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:19:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:19:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:19:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:19:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:20:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:20:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:20:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:20:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:20:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:20:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:20:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:20:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:20:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:20:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:20:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:20:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:20:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:20:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:20:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:20:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:20:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:20:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:20:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:20:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:20:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:20:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:20:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:20:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:20:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:20:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:20:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:20:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:21:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:21:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:21:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:21:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:21:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:21:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:21:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:21:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:21:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:21:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:21:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:21:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:21:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:21:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:21:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:21:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:21:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:21:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:21:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:21:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:21:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:21:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:21:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:21:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:21:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:21:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:21:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:22:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:22:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:22:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:22:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:22:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:22:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:22:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:22:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:22:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:22:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:22:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:22:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:22:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:22:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:22:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:22:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:22:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:22:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:22:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:22:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:22:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:22:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:22:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:22:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:22:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:22:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:22:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:23:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:23:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:23:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:23:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:23:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:23:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:23:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:23:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:23:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:23:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:23:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:23:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:23:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:23:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:23:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:23:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:23:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:23:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:23:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:23:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:23:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:23:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:23:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:23:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:23:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:23:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:23:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:23:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:24:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:24:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:24:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:24:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:24:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:24:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:24:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:24:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:24:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:24:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:24:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:24:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:24:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:24:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:24:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:24:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:24:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:24:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:24:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:24:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:24:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:24:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:24:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:24:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:24:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:24:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:24:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:25:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:25:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:25:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:25:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:25:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:25:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:25:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:25:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:25:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:25:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:25:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:25:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:25:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:25:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:25:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:25:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:25:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:25:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:25:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:25:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:25:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:25:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:25:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:25:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:25:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:25:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:25:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:26:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:26:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:26:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:26:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:26:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:26:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:26:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:26:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:26:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:26:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:26:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:26:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:26:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:26:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:26:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:26:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:26:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:26:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:26:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:26:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:26:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:26:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:26:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:26:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:26:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:26:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:26:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:26:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:27:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:27:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:27:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:27:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:27:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:27:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:27:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:27:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:27:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:27:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:27:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:27:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:27:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:27:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:27:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:27:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:27:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:27:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:27:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:27:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:27:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:27:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:27:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:27:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:27:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:27:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:27:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:28:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:28:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:28:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:28:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:28:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:28:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:28:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:28:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:28:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:28:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:28:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:28:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:28:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:28:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:28:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:28:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:28:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:28:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:28:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:28:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:28:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:28:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:28:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:28:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:28:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:28:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:28:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:29:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:29:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:29:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:29:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:29:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:29:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:29:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:29:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:29:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:29:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:29:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:29:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:29:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:29:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:29:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:29:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:29:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:29:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:29:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:29:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:29:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:29:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:29:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:29:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:29:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:29:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:29:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:29:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:30:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:30:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:30:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:30:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:30:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:30:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:30:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:30:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:30:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:30:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:30:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:30:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:30:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:30:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:30:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:30:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:30:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:30:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:30:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:30:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:30:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:30:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:30:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:30:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:30:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:30:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:30:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:31:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  7 15:31:01 rdma06 dnf[585868]: Updating Subscription Management repositories.
Apr  7 15:31:01 rdma06 dnf[585868]: Unable to read consumer identity
Apr  7 15:31:01 rdma06 dnf[585868]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  7 15:31:01 rdma06 dnf[585868]: Metadata cache refreshed recently.
Apr  7 15:31:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  7 15:31:01 rdma06 systemd[1]: Finished dnf makecache.
Apr  7 15:31:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:31:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:31:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:31:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:31:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:31:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:31:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:31:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:31:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:31:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:31:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:31:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:31:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:31:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:31:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:31:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:31:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:31:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:31:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:31:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:31:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:31:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:31:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:31:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:31:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:31:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:31:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:32:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:32:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:32:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:32:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:32:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:32:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:32:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:32:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:32:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:32:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:32:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:32:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:32:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:32:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:32:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:32:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:32:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:32:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:32:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:32:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:32:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:32:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:32:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:32:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:32:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:32:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:32:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:32:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:33:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:33:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:33:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:33:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:33:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:33:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:33:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:33:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:33:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:33:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:33:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:33:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:33:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:33:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:33:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:33:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:33:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:33:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:33:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:33:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:33:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:33:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:33:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:33:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:33:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:33:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:33:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:34:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:34:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:34:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:34:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:34:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:34:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:34:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:34:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:34:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:34:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:34:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:34:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:34:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:34:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:34:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:34:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:34:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:34:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:34:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:34:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:34:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:34:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:34:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:34:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:34:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:34:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:34:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:35:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:35:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:35:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:35:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:35:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:35:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:35:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:35:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:35:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:35:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:35:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:35:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:35:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:35:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:35:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:35:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:35:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:35:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:35:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:35:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:35:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:35:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:35:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:35:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:35:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:35:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:35:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:36:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:36:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:36:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:36:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:36:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:36:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:36:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:36:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:36:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:36:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:36:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:36:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:36:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:36:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:36:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:36:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:36:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:36:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:36:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:36:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:36:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:36:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:36:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:36:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:36:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:36:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:36:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:36:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:37:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:37:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:37:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:37:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:37:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:37:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:37:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:37:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:37:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:37:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:37:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:37:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:37:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:37:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:37:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:37:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:37:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:37:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:37:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:37:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:37:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:37:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:37:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:37:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:37:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:37:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:37:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:38:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:38:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:38:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:38:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:38:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:38:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:38:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:38:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:38:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:38:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:38:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:38:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:38:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:38:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:38:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:38:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:38:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:38:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:38:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:38:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:38:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:38:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:38:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:38:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:38:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:38:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:38:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:39:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:39:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:39:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:39:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:39:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:39:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:39:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:39:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:39:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:39:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:39:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:39:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:39:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:39:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:39:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:39:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:39:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:39:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:39:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:39:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:39:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:39:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:39:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:39:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:39:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:39:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:39:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:39:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:40:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:40:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:40:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:40:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:40:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:40:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:40:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:40:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:40:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:40:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:40:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:40:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:40:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:40:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:40:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:40:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:40:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:40:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:40:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:40:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:40:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:40:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:40:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:40:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:40:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:40:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:40:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:41:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:41:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:41:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:41:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:41:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:41:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:41:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:41:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:41:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:41:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:41:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:41:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:41:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:41:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:41:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:41:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:41:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:41:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:41:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:41:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:41:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:41:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:41:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:41:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:41:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:41:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:41:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:42:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:42:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:42:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:42:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:42:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:42:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:42:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:42:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:42:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:42:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:42:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:42:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:42:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:42:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:42:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:42:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:42:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:42:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:42:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:42:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:42:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:42:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:42:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:42:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:42:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:42:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:42:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:42:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:43:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:43:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:43:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:43:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:43:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:43:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:43:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:43:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:43:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:43:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:43:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:43:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:43:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:43:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:43:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:43:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:43:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:43:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:43:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:43:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:43:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:43:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:43:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:43:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:43:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:43:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:43:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:44:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:44:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:44:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:44:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:44:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:44:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:44:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:44:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:44:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:44:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:44:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:44:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:44:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:44:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:44:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:44:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:44:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:44:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:44:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:44:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:44:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:44:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:44:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:44:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:44:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:44:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:44:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:45:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:45:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:45:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:45:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:45:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:45:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:45:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:45:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:45:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:45:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:45:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:45:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:45:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:45:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:45:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:45:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:45:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:45:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:45:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:45:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:45:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:45:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:45:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:45:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:45:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:45:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:45:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:45:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:46:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:46:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:46:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:46:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:46:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:46:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:46:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:46:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:46:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:46:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:46:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:46:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:46:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:46:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:46:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:46:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:46:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:46:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:46:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:46:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:46:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:46:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:46:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:46:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:46:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:46:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:46:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:47:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:47:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:47:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:47:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:47:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:47:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:47:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:47:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:47:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:47:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:47:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:47:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:47:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:47:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:47:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:47:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:47:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:47:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:47:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:47:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:47:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:47:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:47:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:47:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:47:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:47:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:47:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:48:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:48:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:48:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:48:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:48:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:48:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:48:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:48:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:48:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:48:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:48:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:48:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:48:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:48:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:48:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:48:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:48:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:48:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:48:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:48:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:48:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:48:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:48:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:48:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:48:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:48:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:48:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:48:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:49:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:49:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:49:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:49:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:49:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:49:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:49:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:49:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:49:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:49:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:49:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:49:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:49:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:49:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:49:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:49:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:49:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:49:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:49:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:49:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:49:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:49:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:49:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:49:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:49:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:49:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:49:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:50:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:50:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:50:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:50:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:50:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:50:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:50:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:50:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:50:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:50:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:50:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:50:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:50:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:50:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:50:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:50:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:50:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:50:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:50:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:50:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:50:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:50:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:50:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:50:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:50:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:50:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:50:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:51:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:51:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:51:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:51:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:51:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:51:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:51:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:51:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:51:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:51:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:51:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:51:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:51:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:51:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:51:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:51:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:51:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:51:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:51:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:51:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:51:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:51:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:51:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:51:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:51:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:51:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:51:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:51:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:52:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:52:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:52:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:52:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:52:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:52:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:52:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:52:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:52:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:52:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:52:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:52:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:52:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:52:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:52:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:52:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:52:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:52:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:52:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:52:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:52:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:52:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:52:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:52:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:52:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:52:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:52:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:53:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:53:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:53:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:53:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:53:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:53:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:53:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:53:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:53:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:53:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:53:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:53:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:53:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:53:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:53:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:53:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:53:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:53:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:53:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:53:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:53:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:53:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:53:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:53:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:53:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:53:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:53:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:54:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:54:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:54:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:54:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:54:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:54:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:54:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:54:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:54:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:54:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:54:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:54:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:54:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:54:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:54:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:54:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:54:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:54:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:54:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:54:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:54:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:54:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:54:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:54:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:54:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:54:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:54:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:55:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:55:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:55:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:55:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:55:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:55:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:55:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:55:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:55:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:55:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:55:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:55:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:55:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:55:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:55:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:55:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:55:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:55:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:55:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:55:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:55:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:55:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:55:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:55:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:55:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:55:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:55:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:55:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:56:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:56:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:56:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:56:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:56:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:56:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:56:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:56:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:56:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:56:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:56:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:56:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:56:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:56:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:56:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:56:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:56:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:56:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:56:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:56:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:56:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:56:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:56:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:56:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:56:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:56:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:56:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:57:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:57:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:57:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:57:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:57:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:57:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:57:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:57:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:57:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:57:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:57:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:57:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:57:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:57:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:57:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:57:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:57:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:57:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:57:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:57:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:57:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:57:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:57:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:57:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:57:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:57:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:57:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:58:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:58:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:58:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:58:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:58:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:58:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:58:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:58:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:58:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:58:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:58:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:58:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:58:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:58:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:58:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:58:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:58:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:58:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:58:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:58:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:58:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:58:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:58:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:58:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:58:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:58:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:58:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:58:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:59:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:59:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:59:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:59:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:59:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:59:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:59:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:59:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:59:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:59:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:59:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:59:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:59:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:59:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:59:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:59:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:59:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:59:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:59:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:59:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:59:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:59:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 15:59:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 15:59:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 15:59:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 15:59:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 15:59:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:00:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:00:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:00:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:00:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:00:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:00:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:00:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:00:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:00:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:00:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:00:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:00:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:00:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:00:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:00:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:00:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:00:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:00:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:00:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:00:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:00:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:00:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:00:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:00:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:00:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:00:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:00:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:01:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:01:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:01:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:01:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:01:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:01:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:01:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:01:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:01:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:01:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:01:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:01:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:01:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:01:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:01:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:01:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:01:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:01:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:01:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:01:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:01:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:01:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:01:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:01:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:01:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:01:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:01:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:01:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:02:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:02:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:02:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:02:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:02:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:02:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:02:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:02:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:02:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:02:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:02:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:02:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:02:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:02:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:02:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:02:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:02:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:02:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:02:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:02:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:02:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:02:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:02:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:02:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:02:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:02:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:02:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:03:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:03:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:03:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:03:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:03:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:03:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:03:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:03:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:03:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:03:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:03:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:03:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:03:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:03:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:03:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:03:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:03:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:03:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:03:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:03:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:03:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:03:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:03:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:03:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:03:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:03:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:03:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:04:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:04:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:04:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:04:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:04:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:04:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:04:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:04:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:04:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:04:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:04:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:04:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:04:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:04:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:04:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:04:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:04:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:04:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:04:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:04:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:04:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:04:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:04:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:04:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:04:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:04:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:04:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:04:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:05:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:05:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:05:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:05:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:05:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:05:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:05:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:05:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:05:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:05:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:05:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:05:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:05:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:05:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:05:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:05:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:05:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:05:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:05:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:05:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:05:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:05:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:05:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:05:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:05:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:05:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:05:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:06:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:06:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:06:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:06:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:06:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:06:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:06:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:06:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:06:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:06:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:06:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:06:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:06:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:06:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:06:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:06:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:06:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:06:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:06:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:06:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:06:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:06:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:06:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:06:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:06:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:06:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:06:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:07:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:07:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:07:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:07:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:07:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:07:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:07:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:07:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:07:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:07:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:07:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:07:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:07:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:07:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:07:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:07:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:07:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:07:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:07:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:07:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:07:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:07:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:07:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:07:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:07:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:07:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:07:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:07:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:08:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:08:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:08:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:08:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:08:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:08:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:08:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:08:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:08:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:08:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:08:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:08:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:08:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:08:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:08:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:08:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:08:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:08:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:08:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:08:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:08:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:08:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:08:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:08:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:08:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:08:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:08:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:09:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:09:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:09:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:09:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:09:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:09:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:09:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:09:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:09:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:09:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:09:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:09:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:09:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:09:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:09:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:09:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:09:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:09:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:09:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:09:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:09:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:09:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:09:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:09:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:09:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:09:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:09:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:10:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:10:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:10:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:10:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:10:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:10:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:10:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:10:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:10:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:10:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:10:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:10:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:10:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:10:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:10:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:10:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:10:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:10:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:10:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:10:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:10:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:10:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:10:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:10:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:10:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:10:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:10:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:11:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:11:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:11:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:11:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:11:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:11:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:11:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:11:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:11:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:11:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:11:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:11:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:11:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:11:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:11:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:11:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:11:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:11:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:11:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:11:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:11:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:11:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:11:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:11:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:11:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:11:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:11:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:11:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:12:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:12:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:12:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:12:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:12:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:12:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:12:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:12:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:12:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:12:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:12:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:12:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:12:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:12:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:12:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:12:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:12:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:12:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:12:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:12:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:12:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:12:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:12:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:12:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:12:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:12:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:12:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:13:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:13:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:13:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:13:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:13:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:13:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:13:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:13:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:13:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:13:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:13:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:13:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:13:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:13:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:13:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:13:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:13:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:13:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:13:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:13:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:13:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:13:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:13:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:13:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:13:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:13:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:13:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:14:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:14:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:14:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:14:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:14:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:14:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:14:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:14:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:14:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:14:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:14:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:14:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:14:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:14:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:14:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:14:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:14:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:14:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:14:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:14:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:14:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:14:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:14:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:14:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:14:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:14:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:14:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:14:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:15:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:15:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:15:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:15:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:15:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:15:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:15:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:15:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:15:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:15:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:15:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:15:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:15:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:15:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:15:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:15:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:15:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:15:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:15:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:15:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:15:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:15:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:15:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:15:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:15:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:15:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:15:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:16:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:16:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:16:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:16:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:16:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:16:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:16:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:16:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:16:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:16:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:16:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:16:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:16:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:16:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:16:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:16:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:16:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:16:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:16:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:16:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:16:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:16:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:16:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:16:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:16:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:16:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:16:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:17:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:17:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:17:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:17:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:17:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:17:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:17:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:17:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:17:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:17:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:17:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:17:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:17:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:17:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:17:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:17:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:17:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:17:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:17:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:17:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:17:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:17:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:17:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:17:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:17:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:17:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:17:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:17:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:18:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:18:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:18:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:18:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:18:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:18:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:18:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:18:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:18:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:18:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:18:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:18:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:18:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:18:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:18:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:18:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:18:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:18:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:18:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:18:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:18:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:18:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:18:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:18:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:18:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:18:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:18:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:19:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:19:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:19:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:19:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:19:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:19:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:19:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:19:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:19:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:19:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:19:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:19:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:19:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:19:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:19:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:19:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:19:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:19:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:19:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:19:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:19:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:19:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:19:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:19:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:19:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:19:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:19:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:20:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:20:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:20:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:20:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:20:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:20:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:20:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:20:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:20:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:20:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:20:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:20:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:20:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:20:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:20:32 rdma06 NetworkManager[1025]: <info>  [1680898832.8645] dhcp4 (eno1): state changed new lease, address=10.73.131.45
Apr  7 16:20:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:20:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:20:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:20:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:20:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:20:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:20:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:20:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:20:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:20:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:20:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:20:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:20:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:20:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:21:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:21:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:21:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:21:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:21:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:21:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:21:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:21:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:21:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:21:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:21:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:21:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:21:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:21:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:21:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:21:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:21:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:21:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:21:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:21:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:21:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:21:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:21:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:21:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:21:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:21:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:21:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:22:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:22:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:22:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:22:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:22:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:22:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:22:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:22:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:22:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:22:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:22:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:22:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:22:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:22:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:22:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:22:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:22:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:22:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:22:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:22:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:22:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:22:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:22:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:22:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:22:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:22:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:22:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:23:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:23:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:23:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:23:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:23:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:23:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:23:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:23:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:23:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:23:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:23:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:23:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:23:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:23:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:23:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:23:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:23:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:23:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:23:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:23:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:23:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:23:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:23:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:23:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:23:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:23:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:23:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:23:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:24:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:24:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:24:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:24:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:24:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:24:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:24:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:24:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:24:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:24:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:24:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:24:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:24:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:24:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:24:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:24:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:24:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:24:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:24:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:24:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:24:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:24:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:24:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:24:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:24:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:24:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:24:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:25:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:25:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:25:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:25:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:25:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:25:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:25:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:25:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:25:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:25:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:25:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:25:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:25:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:25:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:25:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:25:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:25:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:25:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:25:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:25:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:25:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:25:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:25:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:25:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:25:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:25:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:25:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:26:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:26:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:26:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:26:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:26:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:26:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:26:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:26:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:26:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:26:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:26:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:26:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:26:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:26:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:26:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:26:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:26:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:26:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:26:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:26:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:26:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:26:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:26:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:26:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:26:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:26:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:26:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:27:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:27:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:27:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:27:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:27:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:27:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:27:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:27:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:27:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:27:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:27:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:27:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:27:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:27:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:27:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:27:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:27:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:27:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:27:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:27:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:27:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:27:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:27:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:27:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:27:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:27:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:27:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:27:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:28:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:28:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:28:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:28:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:28:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:28:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:28:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:28:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:28:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:28:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:28:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:28:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:28:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:28:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:28:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:28:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:28:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:28:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:28:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:28:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:28:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:28:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:28:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:28:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:28:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:28:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:28:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:29:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:29:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:29:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:29:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:29:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:29:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:29:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:29:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:29:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:29:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:29:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:29:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:29:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:29:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:29:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:29:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:29:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:29:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:29:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:29:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:29:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:29:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:29:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:29:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:29:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:29:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:29:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:30:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:30:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:30:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:30:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:30:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:30:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:30:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:30:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:30:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:30:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:30:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:30:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:30:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:30:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:30:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:30:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:30:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:30:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:30:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:30:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:30:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:30:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:30:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:30:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:30:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:30:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:30:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:30:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:31:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:31:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:31:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:31:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:31:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:31:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:31:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:31:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:31:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:31:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:31:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:31:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:31:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:31:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:31:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:31:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:31:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:31:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:31:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:31:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:31:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:31:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:31:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:31:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:31:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:31:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:31:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:32:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:32:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:32:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:32:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:32:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:32:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:32:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:32:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:32:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:32:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:32:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:32:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:32:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:32:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:32:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:32:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:32:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:32:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:32:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:32:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:32:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:32:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:32:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:32:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:32:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:32:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:32:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:33:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:33:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:33:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:33:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:33:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:33:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:33:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:33:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:33:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:33:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:33:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:33:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:33:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:33:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:33:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:33:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:33:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:33:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:33:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:33:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:33:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:33:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:33:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:33:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:33:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:33:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:33:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:33:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:34:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:34:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:34:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:34:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:34:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:34:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:34:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:34:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:34:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:34:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:34:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:34:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:34:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:34:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:34:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:34:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:34:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:34:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:34:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:34:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:34:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:34:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:34:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:34:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:34:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:34:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:34:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:35:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:35:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:35:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:35:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:35:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:35:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:35:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:35:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:35:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:35:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:35:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:35:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:35:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:35:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:35:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:35:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:35:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:35:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:35:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:35:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:35:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:35:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:35:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:35:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:35:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:35:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:35:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:36:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:36:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:36:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:36:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:36:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:36:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:36:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:36:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:36:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:36:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:36:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:36:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:36:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:36:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:36:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:36:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:36:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:36:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:36:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:36:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:36:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:36:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:36:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:36:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:36:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:36:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:36:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:36:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:37:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:37:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:37:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:37:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:37:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:37:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:37:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:37:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:37:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:37:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:37:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:37:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:37:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:37:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:37:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:37:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:37:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:37:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:37:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:37:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:37:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:37:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:37:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:37:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:37:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:37:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:37:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:38:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:38:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:38:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:38:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:38:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:38:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:38:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:38:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:38:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:38:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:38:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:38:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:38:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:38:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:38:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:38:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:38:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:38:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:38:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:38:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:38:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:38:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:38:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:38:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:38:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:38:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:38:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:39:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:39:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:39:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:39:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:39:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:39:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:39:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:39:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:39:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:39:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:39:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:39:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:39:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:39:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:39:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:39:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:39:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:39:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:39:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:39:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:39:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:39:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:39:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:39:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:39:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:39:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:39:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:39:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:40:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:40:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:40:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:40:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:40:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:40:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:40:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:40:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:40:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:40:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:40:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:40:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:40:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:40:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:40:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:40:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:40:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:40:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:40:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:40:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:40:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:40:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:40:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:40:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:40:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:40:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:40:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:41:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:41:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:41:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:41:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:41:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:41:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:41:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:41:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:41:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:41:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:41:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:41:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:41:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:41:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:41:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:41:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:41:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:41:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:41:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:41:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:41:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:41:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:41:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:41:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:41:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:41:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:41:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:42:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:42:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:42:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:42:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:42:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:42:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:42:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:42:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:42:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:42:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:42:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:42:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:42:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:42:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:42:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:42:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:42:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:42:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:42:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:42:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:42:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:42:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:42:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:42:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:42:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:42:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:42:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:43:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:43:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:43:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:43:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:43:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:43:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:43:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:43:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:43:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:43:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:43:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:43:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:43:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:43:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:43:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:43:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:43:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:43:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:43:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:43:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:43:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:43:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:43:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:43:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:43:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:43:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:43:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:43:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:44:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:44:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:44:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:44:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:44:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:44:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:44:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:44:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:44:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:44:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:44:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:44:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:44:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:44:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:44:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:44:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:44:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:44:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:44:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:44:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:44:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:44:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:44:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:44:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:44:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:44:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:44:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:45:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:45:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:45:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:45:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:45:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:45:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:45:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:45:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:45:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:45:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:45:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:45:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:45:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:45:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:45:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:45:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:45:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:45:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:45:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:45:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:45:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:45:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:45:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:45:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:45:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:45:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:45:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:46:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:46:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:46:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:46:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:46:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:46:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:46:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:46:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:46:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:46:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:46:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:46:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:46:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:46:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:46:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:46:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:46:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:46:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:46:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:46:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:46:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:46:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:46:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:46:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:46:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:46:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:46:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:46:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:47:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:47:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:47:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:47:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:47:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:47:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:47:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:47:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:47:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:47:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:47:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:47:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:47:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:47:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:47:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:47:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:47:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:47:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:47:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:47:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:47:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:47:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:47:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:47:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:47:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:47:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:47:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:48:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:48:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:48:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:48:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:48:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:48:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:48:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:48:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:48:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:48:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:48:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:48:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:48:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:48:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:48:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:48:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:48:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:48:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:48:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:48:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:48:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:48:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:48:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:48:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:48:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:48:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:48:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:49:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:49:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:49:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:49:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:49:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:49:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:49:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:49:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:49:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:49:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:49:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:49:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:49:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:49:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:49:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:49:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:49:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:49:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:49:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:49:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:49:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:49:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:49:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:49:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:49:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:49:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:49:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:49:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:50:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:50:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:50:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:50:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:50:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:50:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:50:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:50:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:50:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:50:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:50:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:50:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:50:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:50:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:50:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:50:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:50:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:50:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:50:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:50:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:50:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:50:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:50:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:50:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:50:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:50:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:50:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:51:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:51:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:51:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:51:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:51:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:51:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:51:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:51:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:51:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:51:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:51:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:51:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:51:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:51:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:51:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:51:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:51:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:51:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:51:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:51:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:51:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:51:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:51:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:51:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:51:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:51:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:51:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:52:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:52:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:52:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:52:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:52:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:52:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:52:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:52:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:52:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:52:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:52:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:52:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:52:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:52:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:52:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:52:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:52:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:52:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:52:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:52:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:52:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:52:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:52:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:52:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:52:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:52:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:52:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:52:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:53:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:53:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:53:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:53:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:53:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:53:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:53:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:53:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:53:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:53:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:53:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:53:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:53:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:53:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:53:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:53:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:53:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:53:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:53:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:53:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:53:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:53:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:53:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:53:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:53:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:53:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:53:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:54:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:54:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:54:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:54:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:54:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:54:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:54:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:54:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:54:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:54:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:54:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:54:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:54:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:54:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:54:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:54:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:54:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:54:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:54:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:54:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:54:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:54:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:54:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:54:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:54:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:54:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:54:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:55:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:55:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:55:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:55:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:55:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:55:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:55:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:55:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:55:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:55:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:55:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:55:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:55:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:55:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:55:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:55:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:55:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:55:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:55:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:55:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:55:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:55:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:55:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:55:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:55:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:55:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:55:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:55:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:56:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:56:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:56:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:56:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:56:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:56:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:56:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:56:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:56:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:56:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:56:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:56:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:56:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:56:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:56:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:56:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:56:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:56:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:56:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:56:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:56:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:56:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:56:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:56:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:56:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:56:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:56:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:57:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:57:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:57:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:57:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:57:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:57:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:57:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:57:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:57:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:57:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:57:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:57:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:57:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:57:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:57:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:57:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:57:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:57:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:57:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:57:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:57:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:57:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:57:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:57:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:57:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:57:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:57:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:58:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:58:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:58:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:58:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:58:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:58:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:58:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:58:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:58:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:58:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:58:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:58:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:58:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:58:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:58:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:58:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:58:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:58:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:58:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:58:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:58:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:58:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:58:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:58:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:58:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:58:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:58:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:58:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:59:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:59:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:59:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:59:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:59:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:59:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:59:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:59:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:59:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:59:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:59:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:59:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:59:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:59:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:59:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:59:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:59:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:59:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:59:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:59:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:59:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:59:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 16:59:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 16:59:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 16:59:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 16:59:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 16:59:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:00:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:00:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:00:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:00:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:00:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:00:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:00:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:00:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:00:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:00:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:00:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:00:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:00:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:00:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:00:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:00:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:00:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:00:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:00:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:00:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:00:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:00:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:00:45 rdma06 systemd[1]: Starting dnf makecache...
Apr  7 17:00:46 rdma06 dnf[589499]: Updating Subscription Management repositories.
Apr  7 17:00:46 rdma06 dnf[589499]: Unable to read consumer identity
Apr  7 17:00:46 rdma06 dnf[589499]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  7 17:00:48 rdma06 dnf[589499]: Copr repo for fmf owned by psss                 2.1 kB/s | 3.3 kB     00:01
Apr  7 17:00:48 rdma06 dnf[589499]: beaker-AppStream                                 49 kB/s | 3.2 kB     00:00
Apr  7 17:00:48 rdma06 dnf[589499]: beaker-AppStream-debuginfo                       40 kB/s | 1.5 kB     00:00
Apr  7 17:00:48 rdma06 dnf[589499]: beaker-BaseOS                                    66 kB/s | 2.7 kB     00:00
Apr  7 17:00:48 rdma06 dnf[589499]: beaker-BaseOS-debuginfo                          47 kB/s | 1.5 kB     00:00
Apr  7 17:00:48 rdma06 dnf[589499]: beaker-CRB                                       67 kB/s | 2.7 kB     00:00
Apr  7 17:00:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:00:48 rdma06 dnf[589499]: beaker-CRB-debuginfo                             68 kB/s | 1.5 kB     00:00
Apr  7 17:00:48 rdma06 dnf[589499]: beaker-HighAvailability                          71 kB/s | 2.8 kB     00:00
Apr  7 17:00:48 rdma06 dnf[589499]: beaker-HighAvailability-debuginfo                26 kB/s | 1.5 kB     00:00
Apr  7 17:00:48 rdma06 dnf[589499]: beaker-NFV                                       51 kB/s | 2.7 kB     00:00
Apr  7 17:00:48 rdma06 dnf[589499]: beaker-NFV-debuginfo                             48 kB/s | 1.5 kB     00:00
Apr  7 17:00:48 rdma06 dnf[589499]: beaker-RT                                        60 kB/s | 2.7 kB     00:00
Apr  7 17:00:49 rdma06 dnf[589499]: beaker-RT-debuginfo                              65 kB/s | 1.5 kB     00:00
Apr  7 17:00:49 rdma06 dnf[589499]: beaker-ResilientStorage                          60 kB/s | 2.8 kB     00:00
Apr  7 17:00:49 rdma06 dnf[589499]: beaker-ResilientStorage-debuginfo                36 kB/s | 1.5 kB     00:00
Apr  7 17:00:49 rdma06 dnf[589499]: beaker-SAP                                       60 kB/s | 2.7 kB     00:00
Apr  7 17:00:49 rdma06 dnf[589499]: beaker-SAP-debuginfo                             48 kB/s | 1.5 kB     00:00
Apr  7 17:00:49 rdma06 dnf[589499]: beaker-SAPHANA                                   84 kB/s | 2.7 kB     00:00
Apr  7 17:00:49 rdma06 dnf[589499]: beaker-SAPHANA-debuginfo                         31 kB/s | 1.5 kB     00:00
Apr  7 17:00:49 rdma06 dnf[589499]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr  7 17:00:50 rdma06 dnf[589499]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr  7 17:00:50 rdma06 dnf[589499]: Metadata cache created.
Apr  7 17:00:50 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  7 17:00:50 rdma06 systemd[1]: Finished dnf makecache.
Apr  7 17:00:50 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.400s CPU time.
Apr  7 17:00:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:00:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:00:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:00:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:01:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:01:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:01:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:01:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:01:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:01:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:01:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:01:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:01:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:01:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:01:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:01:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:01:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:01:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:01:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:01:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:01:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:01:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:01:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:01:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:01:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:01:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:01:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:01:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:01:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:01:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:01:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:02:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:02:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:02:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:02:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:02:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:02:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:02:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:02:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:02:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:02:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:02:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:02:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:02:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:02:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:02:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:02:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:02:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:02:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:02:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:02:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:02:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:02:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:02:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:02:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:02:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:02:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:02:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:02:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:03:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:03:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:03:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:03:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:03:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:03:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:03:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:03:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:03:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:03:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:03:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:03:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:03:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:03:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:03:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:03:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:03:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:03:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:03:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:03:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:03:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:03:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:03:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:03:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:03:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:03:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:03:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:04:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:04:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:04:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:04:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:04:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:04:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:04:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:04:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:04:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:04:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:04:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:04:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:04:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:04:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:04:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:04:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:04:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:04:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:04:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:04:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:04:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:04:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:04:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:04:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:04:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:04:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:04:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:05:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:05:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:05:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:05:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:05:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:05:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:05:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:05:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:05:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:05:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:05:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:05:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:05:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:05:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:05:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:05:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:05:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:05:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:05:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:05:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:05:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:05:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:05:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:05:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:05:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:05:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:05:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:05:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:06:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:06:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:06:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:06:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:06:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:06:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:06:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:06:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:06:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:06:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:06:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:06:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:06:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:06:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:06:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:06:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:06:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:06:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:06:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:06:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:06:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:06:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:06:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:06:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:06:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:06:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:06:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:07:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:07:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:07:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:07:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:07:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:07:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:07:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:07:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:07:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:07:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:07:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:07:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:07:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:07:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:07:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:07:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:07:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:07:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:07:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:07:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:07:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:07:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:07:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:07:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:07:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:07:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:07:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:08:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:08:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:08:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:08:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:08:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:08:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:08:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:08:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:08:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:08:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:08:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:08:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:08:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:08:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:08:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:08:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:08:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:08:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:08:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:08:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:08:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:08:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:08:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:08:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:08:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:08:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:08:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:08:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:09:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:09:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:09:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:09:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:09:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:09:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:09:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:09:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:09:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:09:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:09:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:09:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:09:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:09:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:09:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:09:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:09:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:09:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:09:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:09:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:09:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:09:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:09:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:09:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:09:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:09:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:09:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:10:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:10:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:10:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:10:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:10:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:10:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:10:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:10:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:10:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:10:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:10:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:10:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:10:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:10:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:10:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:10:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:10:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:10:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:10:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:10:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:10:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:10:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:10:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:10:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:10:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:10:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:10:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:11:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:11:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:11:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:11:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:11:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:11:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:11:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:11:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:11:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:11:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:11:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:11:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:11:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:11:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:11:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:11:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:11:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:11:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:11:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:11:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:11:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:11:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:11:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:11:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:11:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:11:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:11:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:11:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:12:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:12:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:12:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:12:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:12:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:12:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:12:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:12:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:12:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:12:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:12:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:12:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:12:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:12:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:12:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:12:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:12:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:12:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:12:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:12:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:12:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:12:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:12:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:12:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:12:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:12:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:12:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:13:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:13:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:13:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:13:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:13:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:13:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:13:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:13:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:13:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:13:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:13:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:13:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:13:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:13:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:13:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:13:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:13:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:13:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:13:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:13:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:13:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:13:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:13:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:13:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:13:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:13:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:13:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:14:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:14:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:14:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:14:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:14:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:14:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:14:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:14:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:14:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:14:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:14:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:14:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:14:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:14:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:14:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:14:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:14:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:14:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:14:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:14:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:14:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:14:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:14:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:14:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:14:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:14:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:14:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:14:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:15:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:15:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:15:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:15:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:15:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:15:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:15:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:15:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:15:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:15:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:15:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:15:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:15:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:15:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:15:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:15:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:15:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:15:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:15:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:15:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:15:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:15:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:15:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:15:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:15:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:15:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:15:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:16:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:16:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:16:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:16:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:16:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:16:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:16:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:16:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:16:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:16:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:16:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:16:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:16:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:16:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:16:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:16:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:16:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:16:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:16:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:16:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:16:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:16:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:16:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:16:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:16:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:16:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:16:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:17:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:17:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:17:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:17:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:17:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:17:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:17:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:17:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:17:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:17:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:17:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:17:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:17:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:17:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:17:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:17:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:17:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:17:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:17:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:17:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:17:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:17:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:17:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:17:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:17:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:17:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:17:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:18:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:18:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:18:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:18:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:18:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:18:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:18:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:18:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:18:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:18:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:18:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:18:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:18:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:18:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:18:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:18:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:18:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:18:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:18:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:18:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:18:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:18:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:18:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:18:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:18:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:18:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:18:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:18:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:19:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:19:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:19:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:19:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:19:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:19:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:19:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:19:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:19:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:19:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:19:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:19:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:19:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:19:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:19:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:19:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:19:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:19:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:19:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:19:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:19:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:19:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:19:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:19:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:19:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:19:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:19:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:20:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:20:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:20:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:20:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:20:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:20:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:20:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:20:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:20:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:20:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:20:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:20:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:20:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:20:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:20:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:20:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:20:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:20:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:20:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:20:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:20:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:20:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:20:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:20:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:20:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:20:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:20:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:21:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:21:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:21:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:21:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:21:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:21:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:21:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:21:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:21:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:21:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:21:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:21:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:21:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:21:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:21:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:21:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:21:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:21:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:21:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:21:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:21:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:21:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:21:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:21:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:21:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:21:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:21:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:21:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:22:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:22:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:22:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:22:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:22:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:22:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:22:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:22:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:22:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:22:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:22:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:22:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:22:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:22:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:22:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:22:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:22:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:22:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:22:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:22:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:22:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:22:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:22:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:22:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:22:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:22:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:22:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:23:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:23:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:23:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:23:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:23:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:23:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:23:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:23:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:23:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:23:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:23:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:23:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:23:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:23:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:23:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:23:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:23:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:23:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:23:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:23:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:23:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:23:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:23:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:23:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:23:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:23:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:23:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:24:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:24:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:24:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:24:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:24:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:24:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:24:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:24:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:24:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:24:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:24:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:24:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:24:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:24:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:24:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:24:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:24:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:24:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:24:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:24:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:24:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:24:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:24:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:24:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:24:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:24:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:24:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:24:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:25:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:25:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:25:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:25:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:25:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:25:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:25:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:25:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:25:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:25:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:25:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:25:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:25:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:25:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:25:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:25:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:25:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:25:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:25:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:25:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:25:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:25:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:25:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:25:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:25:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:25:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:25:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:26:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:26:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:26:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:26:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:26:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:26:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:26:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:26:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:26:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:26:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:26:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:26:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:26:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:26:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:26:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:26:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:26:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:26:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:26:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:26:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:26:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:26:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:26:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:26:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:26:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:26:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:26:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:27:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:27:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:27:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:27:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:27:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:27:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:27:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:27:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:27:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:27:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:27:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:27:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:27:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:27:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:27:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:27:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:27:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:27:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:27:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:27:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:27:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:27:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:27:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:27:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:27:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:27:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:27:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:27:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:28:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:28:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:28:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:28:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:28:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:28:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:28:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:28:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:28:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:28:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:28:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:28:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:28:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:28:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:28:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:28:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:28:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:28:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:28:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:28:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:28:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:28:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:28:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:28:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:28:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:28:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:28:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:29:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:29:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:29:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:29:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:29:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:29:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:29:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:29:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:29:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:29:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:29:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:29:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:29:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:29:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:29:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:29:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:29:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:29:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:29:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:29:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:29:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:29:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:29:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:29:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:29:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:29:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:29:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:30:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:30:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:30:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:30:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:30:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:30:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:30:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:30:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:30:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:30:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:30:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:30:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:30:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:30:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:30:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:30:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:30:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:30:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:30:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:30:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:30:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:30:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:30:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:30:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:30:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:30:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:30:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:30:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:31:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:31:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:31:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:31:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:31:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:31:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:31:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:31:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:31:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:31:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:31:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:31:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:31:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:31:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:31:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:31:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:31:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:31:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:31:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:31:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:31:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:31:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:31:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:31:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:31:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:31:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:31:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:32:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:32:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:32:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:32:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:32:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:32:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:32:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:32:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:32:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:32:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:32:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:32:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:32:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:32:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:32:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:32:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:32:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:32:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:32:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:32:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:32:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:32:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:32:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:32:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:32:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:32:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:32:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:33:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:33:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:33:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:33:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:33:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:33:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:33:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:33:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:33:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:33:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:33:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:33:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:33:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:33:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:33:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:33:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:33:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:33:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:33:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:33:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:33:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:33:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:33:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:33:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:33:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:33:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:33:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:34:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:34:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:34:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:34:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:34:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:34:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:34:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:34:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:34:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:34:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:34:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:34:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:34:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:34:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:34:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:34:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:34:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:34:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:34:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:34:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:34:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:34:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:34:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:34:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:34:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:34:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:34:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:34:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:35:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:35:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:35:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:35:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:35:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:35:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:35:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:35:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:35:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:35:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:35:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:35:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:35:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:35:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:35:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:35:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:35:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:35:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:35:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:35:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:35:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:35:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:35:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:35:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:35:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:35:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:35:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:36:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:36:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:36:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:36:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:36:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:36:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:36:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:36:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:36:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:36:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:36:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:36:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:36:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:36:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:36:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:36:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:36:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:36:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:36:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:36:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:36:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:36:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:36:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:36:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:36:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:36:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:36:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:37:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:37:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:37:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:37:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:37:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:37:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:37:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:37:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:37:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:37:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:37:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:37:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:37:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:37:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:37:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:37:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:37:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:37:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:37:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:37:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:37:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:37:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:37:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:37:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:37:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:37:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:37:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:37:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:38:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:38:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:38:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:38:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:38:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:38:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:38:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:38:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:38:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:38:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:38:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:38:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:38:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:38:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:38:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:38:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:38:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:38:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:38:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:38:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:38:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:38:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:38:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:38:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:38:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:38:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:38:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:39:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:39:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:39:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:39:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:39:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:39:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:39:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:39:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:39:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:39:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:39:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:39:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:39:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:39:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:39:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:39:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:39:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:39:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:39:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:39:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:39:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:39:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:39:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:39:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:39:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:39:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:39:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:40:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:40:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:40:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:40:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:40:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:40:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:40:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:40:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:40:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:40:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:40:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:40:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:40:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:40:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:40:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:40:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:40:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:40:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:40:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:40:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:40:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:40:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:40:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:40:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:40:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:40:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:40:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:40:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:41:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:41:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:41:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:41:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:41:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:41:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:41:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:41:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:41:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:41:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:41:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:41:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:41:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:41:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:41:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:41:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:41:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:41:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:41:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:41:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:41:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:41:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:41:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:41:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:41:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:41:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:41:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:42:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:42:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:42:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:42:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:42:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:42:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:42:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:42:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:42:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:42:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:42:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:42:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:42:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:42:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:42:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:42:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:42:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:42:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:42:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:42:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:42:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:42:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:42:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:42:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:42:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:42:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:42:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:43:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:43:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:43:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:43:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:43:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:43:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:43:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:43:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:43:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:43:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:43:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:43:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:43:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:43:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:43:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:43:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:43:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:43:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:43:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:43:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:43:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:43:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:43:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:43:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:43:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:43:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:43:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:43:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:44:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:44:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:44:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:44:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:44:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:44:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:44:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:44:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:44:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:44:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:44:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:44:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:44:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:44:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:44:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:44:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:44:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:44:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:44:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:44:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:44:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:44:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:44:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:44:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:44:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:44:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:44:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:45:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:45:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:45:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:45:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:45:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:45:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:45:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:45:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:45:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:45:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:45:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:45:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:45:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:45:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:45:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:45:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:45:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:45:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:45:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:45:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:45:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:45:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:45:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:45:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:45:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:45:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:45:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:46:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:46:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:46:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:46:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:46:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:46:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:46:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:46:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:46:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:46:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:46:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:46:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:46:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:46:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:46:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:46:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:46:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:46:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:46:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:46:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:46:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:46:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:46:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:46:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:46:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:46:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:46:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:46:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:47:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:47:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:47:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:47:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:47:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:47:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:47:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:47:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:47:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:47:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:47:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:47:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:47:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:47:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:47:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:47:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:47:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:47:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:47:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:47:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:47:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:47:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:47:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:47:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:47:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:47:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:47:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:48:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:48:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:48:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:48:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:48:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:48:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:48:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:48:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:48:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:48:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:48:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:48:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:48:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:48:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:48:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:48:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:48:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:48:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:48:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:48:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:48:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:48:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:48:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:48:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:48:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:48:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:48:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:49:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:49:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:49:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:49:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:49:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:49:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:49:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:49:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:49:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:49:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:49:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:49:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:49:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:49:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:49:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:49:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:49:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:49:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:49:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:49:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:49:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:49:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:49:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:49:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:49:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:49:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:49:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:50:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:50:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:50:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:50:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:50:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:50:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:50:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:50:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:50:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:50:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:50:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:50:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:50:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:50:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:50:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:50:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:50:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:50:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:50:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:50:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:50:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:50:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:50:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:50:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:50:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:50:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:50:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:50:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:51:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:51:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:51:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:51:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:51:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:51:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:51:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:51:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:51:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:51:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:51:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:51:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:51:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:51:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:51:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:51:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:51:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:51:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:51:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:51:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:51:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:51:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:51:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:51:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:51:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:51:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:51:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:52:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:52:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:52:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:52:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:52:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:52:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:52:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:52:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:52:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:52:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:52:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:52:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:52:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:52:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:52:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:52:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:52:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:52:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:52:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:52:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:52:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:52:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:52:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:52:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:52:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:52:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:52:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:53:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:53:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:53:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:53:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:53:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:53:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:53:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:53:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:53:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:53:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:53:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:53:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:53:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:53:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:53:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:53:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:53:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:53:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:53:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:53:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:53:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:53:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:53:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:53:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:53:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:53:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:53:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:53:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:54:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:54:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:54:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:54:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:54:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:54:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:54:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:54:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:54:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:54:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:54:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:54:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:54:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:54:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:54:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:54:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:54:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:54:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:54:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:54:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:54:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:54:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:54:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:54:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:54:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:54:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:54:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:55:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:55:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:55:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:55:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:55:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:55:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:55:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:55:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:55:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:55:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:55:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:55:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:55:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:55:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:55:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:55:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:55:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:55:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:55:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:55:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:55:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:55:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:55:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:55:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:55:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:55:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:55:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:56:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:56:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:56:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:56:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:56:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:56:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:56:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:56:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:56:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:56:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:56:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:56:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:56:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:56:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:56:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:56:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:56:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:56:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:56:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:56:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:56:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:56:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:56:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:56:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:56:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:56:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:56:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:56:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:57:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:57:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:57:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:57:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:57:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:57:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:57:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:57:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:57:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:57:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:57:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:57:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:57:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:57:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:57:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:57:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:57:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:57:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:57:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:57:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:57:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:57:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:57:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:57:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:57:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:57:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:57:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:58:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:58:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:58:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:58:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:58:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:58:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:58:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:58:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:58:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:58:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:58:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:58:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:58:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:58:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:58:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:58:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:58:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:58:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:58:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:58:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:58:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:58:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:58:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:58:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:58:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:58:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:58:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:59:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:59:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:59:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:59:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:59:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:59:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:59:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:59:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:59:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:59:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:59:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:59:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:59:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:59:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:59:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:59:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:59:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:59:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:59:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 17:59:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:59:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:59:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:59:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:59:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 17:59:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 17:59:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 17:59:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 17:59:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:00:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:00:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:00:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:00:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:00:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:00:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:00:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:00:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:00:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:00:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:00:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:00:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:00:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:00:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:00:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:00:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:00:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:00:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:00:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:00:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:00:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:00:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:00:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:00:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:00:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:00:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:00:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:01:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:01:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:01:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:01:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:01:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:01:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:01:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:01:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:01:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:01:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:01:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:01:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:01:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:01:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:01:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:01:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:01:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:01:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:01:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:01:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:01:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:01:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:01:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:01:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:01:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:01:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:01:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:02:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:02:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:02:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:02:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:02:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:02:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:02:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:02:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:02:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:02:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:02:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:02:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:02:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:02:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:02:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:02:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:02:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:02:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:02:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:02:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:02:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:02:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:02:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:02:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:02:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:02:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:02:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:02:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:03:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:03:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:03:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:03:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:03:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:03:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:03:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:03:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:03:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:03:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:03:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:03:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:03:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:03:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:03:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:03:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:03:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:03:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:03:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:03:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:03:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:03:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:03:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:03:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:03:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:03:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:03:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:04:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:04:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:04:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:04:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:04:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:04:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:04:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:04:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:04:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:04:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:04:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:04:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:04:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:04:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:04:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:04:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:04:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:04:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:04:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:04:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:04:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:04:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:04:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:04:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:04:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:04:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:04:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:05:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:05:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:05:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:05:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:05:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:05:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:05:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:05:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:05:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:05:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:05:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:05:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:05:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:05:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:05:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:05:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:05:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:05:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:05:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:05:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:05:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:05:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:05:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:05:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:05:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:05:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:05:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:05:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:06:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:06:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:06:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:06:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:06:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:06:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:06:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:06:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:06:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:06:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:06:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:06:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:06:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:06:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:06:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:06:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:06:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:06:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:06:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:06:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:06:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:06:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:06:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:06:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:06:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:06:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:06:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:07:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:07:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:07:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:07:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:07:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:07:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:07:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:07:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:07:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:07:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:07:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:07:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:07:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:07:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:07:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:07:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:07:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:07:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:07:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:07:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:07:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:07:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:07:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:07:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:07:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:07:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:07:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:08:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:08:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:08:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:08:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:08:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:08:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:08:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:08:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:08:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:08:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:08:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:08:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:08:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:08:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:08:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:08:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:08:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:08:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:08:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:08:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:08:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:08:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:08:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:08:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:08:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:08:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:08:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:09:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:09:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:09:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:09:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:09:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:09:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:09:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:09:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:09:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:09:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:09:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:09:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:09:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:09:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:09:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:09:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:09:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:09:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:09:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:09:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:09:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:09:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:09:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:09:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:09:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:09:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:09:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:09:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:10:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:10:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:10:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:10:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:10:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:10:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:10:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:10:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:10:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:10:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:10:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:10:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:10:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:10:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:10:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:10:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:10:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:10:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:10:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:10:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:10:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:10:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:10:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:10:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:10:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:10:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:10:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:11:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:11:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:11:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:11:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:11:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:11:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:11:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:11:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:11:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:11:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:11:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:11:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:11:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:11:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:11:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:11:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:11:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:11:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:11:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:11:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:11:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:11:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:11:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:11:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:11:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:11:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:11:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:12:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:12:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:12:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:12:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:12:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:12:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:12:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:12:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:12:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:12:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:12:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:12:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:12:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:12:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:12:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:12:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:12:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:12:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:12:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:12:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:12:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:12:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:12:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:12:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:12:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:12:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:12:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:12:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:13:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:13:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:13:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:13:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:13:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:13:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:13:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:13:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:13:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:13:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:13:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:13:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:13:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:13:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:13:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:13:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:13:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:13:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:13:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:13:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:13:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:13:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:13:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:13:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:13:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:13:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:13:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:14:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:14:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:14:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:14:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:14:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:14:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:14:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:14:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:14:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:14:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:14:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:14:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:14:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:14:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:14:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:14:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:14:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:14:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:14:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:14:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:14:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:14:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:14:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:14:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:14:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:14:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:14:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:15:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:15:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:15:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:15:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:15:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:15:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:15:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:15:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:15:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:15:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:15:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:15:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:15:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:15:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:15:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:15:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:15:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:15:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:15:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:15:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:15:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:15:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:15:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:15:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:15:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:15:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:15:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:15:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:16:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:16:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:16:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:16:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:16:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:16:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:16:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:16:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:16:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:16:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:16:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:16:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:16:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:16:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:16:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:16:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:16:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:16:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:16:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:16:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:16:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:16:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:16:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:16:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:16:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:16:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:16:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:17:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:17:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:17:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:17:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:17:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:17:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:17:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:17:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:17:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:17:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:17:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:17:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:17:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:17:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:17:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:17:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:17:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:17:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:17:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:17:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:17:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:17:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:17:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:17:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:17:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:17:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:17:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:18:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:18:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:18:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:18:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:18:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:18:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:18:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:18:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:18:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:18:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:18:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:18:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:18:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:18:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:18:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:18:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:18:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:18:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:18:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:18:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:18:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:18:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:18:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:18:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:18:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:18:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:18:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:18:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:19:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:19:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:19:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:19:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:19:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:19:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:19:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:19:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:19:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:19:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:19:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:19:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:19:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:19:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:19:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:19:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:19:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:19:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:19:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:19:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:19:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:19:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:19:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:19:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:19:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:19:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:19:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:20:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:20:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:20:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:20:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:20:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:20:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:20:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:20:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:20:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:20:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:20:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:20:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:20:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:20:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:20:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:20:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:20:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:20:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:20:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:20:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:20:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:20:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:20:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:20:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:20:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:20:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:20:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:21:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:21:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:21:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:21:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:21:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:21:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:21:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:21:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:21:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:21:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:21:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:21:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:21:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:21:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:21:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:21:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:21:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:21:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:21:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:21:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:21:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:21:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:21:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:21:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:21:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:21:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:21:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:21:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:22:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:22:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:22:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:22:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:22:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:22:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:22:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:22:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:22:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:22:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:22:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:22:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:22:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:22:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:22:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:22:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:22:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:22:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:22:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:22:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:22:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:22:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:22:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:22:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:22:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:22:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:22:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:23:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:23:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:23:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:23:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:23:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:23:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:23:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:23:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:23:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:23:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:23:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:23:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:23:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:23:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:23:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:23:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:23:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:23:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:23:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:23:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:23:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:23:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:23:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:23:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:23:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:23:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:23:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:24:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:24:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:24:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:24:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:24:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:24:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:24:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:24:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:24:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:24:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:24:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:24:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:24:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:24:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:24:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:24:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:24:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:24:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:24:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:24:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:24:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:24:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:24:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:24:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:24:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:24:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:24:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:25:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:25:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:25:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:25:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:25:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:25:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:25:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:25:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:25:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:25:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:25:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:25:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:25:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:25:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:25:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:25:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:25:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:25:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:25:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:25:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:25:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:25:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:25:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:25:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:25:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:25:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:25:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:25:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:26:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:26:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:26:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:26:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:26:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:26:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:26:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:26:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:26:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:26:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:26:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:26:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:26:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:26:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:26:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:26:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:26:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:26:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:26:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:26:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:26:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:26:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:26:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:26:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:26:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:26:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:26:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:27:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:27:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:27:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:27:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:27:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:27:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:27:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:27:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:27:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:27:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:27:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:27:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:27:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:27:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:27:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:27:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:27:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:27:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:27:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:27:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:27:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:27:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:27:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:27:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:27:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:27:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:27:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:28:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:28:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:28:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:28:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:28:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:28:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:28:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:28:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:28:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:28:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:28:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:28:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:28:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:28:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:28:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:28:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:28:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:28:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:28:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:28:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:28:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:28:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:28:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:28:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:28:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:28:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:28:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:28:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:29:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:29:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:29:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:29:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:29:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:29:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:29:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:29:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:29:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:29:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:29:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:29:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:29:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:29:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:29:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:29:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:29:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:29:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:29:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:29:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:29:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:29:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:29:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:29:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:29:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:29:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:29:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:30:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:30:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:30:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:30:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:30:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:30:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:30:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:30:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:30:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:30:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:30:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:30:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:30:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:30:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:30:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:30:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:30:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:30:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:30:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:30:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:30:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:30:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:30:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:30:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:30:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:30:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:30:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:31:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:31:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:31:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:31:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:31:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:31:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:31:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:31:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:31:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:31:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:31:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:31:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:31:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:31:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:31:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:31:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:31:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:31:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:31:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:31:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:31:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:31:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:31:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:31:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:31:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:31:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:31:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:31:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:32:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:32:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:32:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:32:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:32:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:32:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:32:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:32:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:32:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:32:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:32:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:32:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:32:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:32:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:32:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:32:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:32:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:32:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:32:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:32:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:32:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:32:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:32:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:32:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:32:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:32:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:32:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:33:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:33:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:33:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:33:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:33:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:33:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:33:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:33:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:33:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:33:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:33:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:33:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:33:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:33:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:33:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:33:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:33:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:33:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:33:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:33:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:33:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:33:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:33:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:33:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:33:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:33:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:33:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:34:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:34:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:34:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:34:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:34:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:34:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:34:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:34:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:34:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:34:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:34:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:34:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:34:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:34:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:34:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:34:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:34:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:34:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:34:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:34:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:34:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:34:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:34:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:34:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:34:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:34:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:34:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:34:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:35:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:35:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:35:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:35:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:35:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:35:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:35:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:35:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:35:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:35:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:35:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:35:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:35:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:35:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:35:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:35:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:35:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:35:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:35:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:35:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:35:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:35:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:35:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:35:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:35:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:35:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:35:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:36:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:36:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:36:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:36:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:36:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:36:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:36:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:36:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:36:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:36:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:36:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:36:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:36:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:36:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:36:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:36:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:36:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:36:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:36:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:36:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:36:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:36:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:36:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:36:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:36:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:36:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:36:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:37:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:37:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:37:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:37:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:37:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:37:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:37:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:37:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:37:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:37:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:37:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:37:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:37:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:37:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:37:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:37:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:37:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:37:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:37:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:37:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:37:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:37:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:37:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:37:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:37:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:37:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:37:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:37:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:38:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:38:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:38:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:38:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:38:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:38:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:38:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:38:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:38:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:38:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:38:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:38:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:38:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:38:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:38:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:38:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:38:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:38:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:38:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:38:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:38:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:38:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:38:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:38:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:38:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:38:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:38:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:39:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:39:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:39:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:39:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:39:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:39:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:39:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:39:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:39:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:39:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:39:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:39:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:39:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:39:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:39:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:39:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:39:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:39:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:39:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:39:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:39:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:39:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:39:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:39:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:39:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:39:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:39:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:40:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:40:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:40:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:40:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:40:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:40:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:40:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:40:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:40:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:40:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:40:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:40:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:40:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:40:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:40:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:40:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:40:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:40:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:40:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:40:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:40:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:40:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:40:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:40:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:40:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:40:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:40:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:41:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:41:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:41:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:41:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:41:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:41:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:41:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:41:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:41:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:41:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:41:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:41:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:41:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:41:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:41:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:41:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:41:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:41:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:41:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:41:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:41:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:41:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:41:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:41:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:41:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:41:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:41:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:41:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:42:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:42:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:42:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:42:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:42:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:42:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:42:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:42:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:42:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:42:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:42:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:42:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:42:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:42:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:42:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:42:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:42:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:42:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:42:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:42:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:42:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:42:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:42:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:42:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:42:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:42:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:42:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:43:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:43:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:43:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:43:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:43:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:43:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:43:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:43:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:43:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:43:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:43:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:43:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:43:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:43:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:43:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:43:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:43:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:43:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:43:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:43:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:43:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:43:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:43:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:43:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:43:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:43:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:43:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:44:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:44:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:44:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:44:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:44:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:44:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:44:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:44:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:44:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:44:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:44:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:44:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:44:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:44:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:44:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:44:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:44:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:44:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:44:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:44:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:44:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:44:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:44:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:44:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:44:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:44:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:44:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:44:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:45:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:45:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:45:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:45:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:45:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:45:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:45:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:45:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:45:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:45:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:45:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:45:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:45:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:45:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:45:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:45:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:45:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:45:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:45:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:45:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:45:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:45:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:45:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:45:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:45:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:45:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:45:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:46:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:46:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:46:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:46:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:46:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:46:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:46:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:46:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:46:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:46:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:46:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:46:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:46:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:46:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:46:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:46:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:46:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:46:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:46:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:46:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:46:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:46:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:46:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:46:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:46:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:46:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:46:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:47:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:47:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:47:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:47:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:47:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:47:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:47:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:47:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:47:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:47:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:47:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:47:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:47:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:47:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:47:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:47:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:47:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:47:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:47:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:47:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:47:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:47:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:47:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:47:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:47:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:47:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:47:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:47:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:48:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:48:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:48:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:48:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:48:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:48:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:48:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:48:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:48:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:48:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:48:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:48:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:48:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:48:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:48:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:48:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:48:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:48:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:48:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:48:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:48:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:48:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:48:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:48:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:48:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:48:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:48:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:49:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:49:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:49:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:49:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:49:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:49:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:49:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:49:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:49:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:49:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:49:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:49:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:49:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:49:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:49:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:49:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:49:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:49:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:49:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:49:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:49:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:49:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:49:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:49:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:49:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:49:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:49:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:50:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:50:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:50:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:50:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:50:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:50:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:50:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:50:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:50:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:50:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:50:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:50:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:50:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:50:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:50:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:50:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:50:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:50:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:50:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:50:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:50:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:50:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:50:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:50:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:50:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:50:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:50:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:50:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:51:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:51:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:51:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:51:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:51:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:51:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:51:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:51:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:51:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:51:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:51:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:51:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:51:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:51:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:51:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:51:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:51:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:51:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:51:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:51:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:51:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:51:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:51:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:51:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:51:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:51:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:51:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:52:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:52:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:52:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:52:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:52:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:52:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:52:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:52:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:52:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:52:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:52:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:52:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:52:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:52:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:52:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:52:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:52:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:52:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:52:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:52:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:52:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:52:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:52:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:52:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:52:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:52:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:52:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:53:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:53:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:53:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:53:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:53:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:53:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:53:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:53:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:53:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:53:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:53:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:53:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:53:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:53:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:53:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:53:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:53:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:53:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:53:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:53:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:53:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:53:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:53:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:53:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:53:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:53:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:53:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:53:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:54:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  7 18:54:01 rdma06 dnf[594100]: Updating Subscription Management repositories.
Apr  7 18:54:01 rdma06 dnf[594100]: Unable to read consumer identity
Apr  7 18:54:01 rdma06 dnf[594100]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  7 18:54:01 rdma06 dnf[594100]: Metadata cache refreshed recently.
Apr  7 18:54:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  7 18:54:01 rdma06 systemd[1]: Finished dnf makecache.
Apr  7 18:54:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:54:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:54:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:54:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:54:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:54:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:54:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:54:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:54:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:54:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:54:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:54:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:54:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:54:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:54:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:54:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:54:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:54:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:54:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:54:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:54:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:54:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:54:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:54:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:54:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:54:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:54:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:55:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:55:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:55:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:55:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:55:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:55:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:55:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:55:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:55:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:55:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:55:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:55:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:55:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:55:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:55:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:55:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:55:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:55:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:55:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:55:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:55:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:55:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:55:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:55:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:55:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:55:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:55:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:56:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:56:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:56:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:56:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:56:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:56:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:56:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:56:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:56:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:56:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:56:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:56:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:56:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:56:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:56:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:56:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:56:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:56:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:56:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:56:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:56:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:56:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:56:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:56:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:56:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:56:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:56:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:57:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:57:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:57:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:57:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:57:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:57:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:57:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:57:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:57:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:57:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:57:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:57:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:57:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:57:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:57:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:57:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:57:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:57:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:57:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:57:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:57:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:57:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:57:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:57:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:57:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:57:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:57:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:57:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:58:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:58:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:58:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:58:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:58:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:58:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:58:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:58:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:58:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:58:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:58:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:58:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:58:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:58:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:58:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:58:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:58:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:58:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:58:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:58:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:58:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:58:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:58:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:58:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:58:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:58:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:58:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:59:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:59:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:59:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:59:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:59:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:59:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:59:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:59:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:59:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:59:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:59:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:59:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:59:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:59:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:59:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:59:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:59:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:59:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:59:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 18:59:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:59:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:59:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:59:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 18:59:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 18:59:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 18:59:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 18:59:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:00:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:00:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:00:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:00:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:00:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:00:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:00:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:00:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:00:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:00:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:00:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:00:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:00:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:00:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:00:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:00:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:00:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:00:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:00:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:00:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:00:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:00:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:00:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:00:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:00:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:00:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:00:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:00:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:01:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:01:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:01:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:01:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:01:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:01:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:01:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:01:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:01:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:01:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:01:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:01:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:01:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:01:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:01:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:01:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:01:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:01:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:01:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:01:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:01:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:01:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:01:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:01:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:01:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:01:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:01:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:02:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:02:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:02:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:02:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:02:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:02:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:02:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:02:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:02:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:02:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:02:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:02:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:02:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:02:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:02:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:02:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:02:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:02:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:02:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:02:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:02:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:02:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:02:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:02:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:02:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:02:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:02:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:03:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:03:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:03:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:03:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:03:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:03:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:03:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:03:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:03:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:03:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:03:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:03:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:03:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:03:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:03:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:03:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:03:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:03:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:03:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:03:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:03:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:03:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:03:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:03:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:03:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:03:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:03:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:03:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:04:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:04:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:04:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:04:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:04:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:04:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:04:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:04:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:04:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:04:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:04:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:04:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:04:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:04:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:04:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:04:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:04:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:04:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:04:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:04:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:04:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:04:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:04:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:04:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:04:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:04:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:04:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:05:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:05:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:05:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:05:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:05:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:05:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:05:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:05:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:05:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:05:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:05:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:05:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:05:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:05:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:05:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:05:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:05:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:05:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:05:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:05:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:05:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:05:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:05:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:05:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:05:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:05:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:05:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:06:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:06:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:06:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:06:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:06:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:06:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:06:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:06:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:06:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:06:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:06:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:06:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:06:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:06:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:06:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:06:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:06:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:06:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:06:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:06:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:06:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:06:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:06:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:06:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:06:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:06:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:06:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:06:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:07:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:07:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:07:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:07:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:07:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:07:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:07:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:07:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:07:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:07:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:07:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:07:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:07:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:07:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:07:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:07:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:07:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:07:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:07:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:07:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:07:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:07:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:07:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:07:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:07:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:07:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:07:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:08:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:08:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:08:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:08:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:08:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:08:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:08:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:08:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:08:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:08:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:08:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:08:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:08:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:08:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:08:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:08:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:08:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:08:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:08:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:08:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:08:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:08:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:08:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:08:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:08:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:08:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:08:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:09:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:09:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:09:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:09:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:09:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:09:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:09:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:09:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:09:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:09:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:09:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:09:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:09:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:09:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:09:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:09:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:09:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:09:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:09:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:09:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:09:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:09:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:09:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:09:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:09:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:09:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:09:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:09:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:10:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:10:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:10:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:10:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:10:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:10:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:10:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:10:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:10:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:10:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:10:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:10:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:10:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:10:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:10:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:10:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:10:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:10:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:10:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:10:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:10:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:10:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:10:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:10:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:10:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:10:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:10:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:11:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:11:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:11:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:11:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:11:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:11:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:11:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:11:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:11:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:11:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:11:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:11:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:11:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:11:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:11:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:11:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:11:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:11:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:11:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:11:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:11:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:11:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:11:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:11:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:11:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:11:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:11:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:12:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:12:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:12:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:12:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:12:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:12:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:12:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:12:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:12:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:12:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:12:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:12:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:12:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:12:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:12:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:12:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:12:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:12:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:12:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:12:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:12:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:12:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:12:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:12:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:12:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:12:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:12:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:12:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:13:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:13:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:13:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:13:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:13:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:13:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:13:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:13:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:13:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:13:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:13:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:13:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:13:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:13:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:13:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:13:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:13:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:13:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:13:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:13:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:13:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:13:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:13:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:13:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:13:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:13:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:13:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:14:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:14:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:14:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:14:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:14:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:14:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:14:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:14:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:14:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:14:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:14:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:14:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:14:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:14:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:14:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:14:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:14:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:14:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:14:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:14:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:14:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:14:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:14:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:14:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:14:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:14:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:14:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:15:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:15:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:15:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:15:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:15:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:15:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:15:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:15:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:15:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:15:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:15:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:15:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:15:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:15:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:15:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:15:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:15:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:15:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:15:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:15:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:15:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:15:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:15:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:15:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:15:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:15:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:15:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:16:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:16:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:16:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:16:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:16:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:16:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:16:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:16:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:16:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:16:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:16:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:16:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:16:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:16:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:16:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:16:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:16:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:16:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:16:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:16:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:16:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:16:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:16:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:16:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:16:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:16:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:16:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:16:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:17:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:17:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:17:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:17:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:17:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:17:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:17:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:17:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:17:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:17:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:17:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:17:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:17:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:17:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:17:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:17:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:17:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:17:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:17:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:17:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:17:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:17:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:17:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:17:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:17:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:17:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:17:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:18:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:18:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:18:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:18:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:18:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:18:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:18:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:18:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:18:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:18:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:18:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:18:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:18:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:18:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:18:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:18:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:18:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:18:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:18:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:18:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:18:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:18:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:18:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:18:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:18:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:18:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:18:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:19:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:19:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:19:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:19:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:19:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:19:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:19:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:19:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:19:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:19:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:19:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:19:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:19:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:19:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:19:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:19:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:19:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:19:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:19:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:19:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:19:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:19:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:19:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:19:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:19:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:19:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:19:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:19:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:20:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:20:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:20:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:20:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:20:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:20:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:20:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:20:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:20:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:20:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:20:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:20:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:20:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:20:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:20:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:20:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:20:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:20:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:20:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:20:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:20:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:20:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:20:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:20:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:20:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:20:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:20:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:21:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:21:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:21:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:21:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:21:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:21:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:21:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:21:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:21:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:21:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:21:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:21:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:21:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:21:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:21:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:21:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:21:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:21:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:21:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:21:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:21:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:21:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:21:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:21:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:21:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:21:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:21:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:22:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:22:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:22:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:22:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:22:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:22:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:22:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:22:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:22:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:22:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:22:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:22:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:22:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:22:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:22:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:22:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:22:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:22:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:22:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:22:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:22:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:22:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:22:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:22:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:22:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:22:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:22:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:22:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:23:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:23:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:23:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:23:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:23:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:23:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:23:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:23:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:23:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:23:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:23:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:23:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:23:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:23:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:23:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:23:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:23:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:23:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:23:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:23:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:23:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:23:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:23:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:23:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:23:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:23:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:23:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:24:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:24:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:24:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:24:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:24:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:24:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:24:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:24:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:24:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:24:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:24:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:24:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:24:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:24:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:24:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:24:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:24:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:24:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:24:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:24:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:24:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:24:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:24:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:24:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:24:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:24:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:24:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:25:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:25:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:25:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:25:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:25:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:25:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:25:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:25:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:25:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:25:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:25:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:25:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:25:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:25:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:25:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:25:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:25:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:25:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:25:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:25:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:25:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:25:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:25:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:25:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:25:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:25:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:25:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:25:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:26:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:26:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:26:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:26:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:26:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:26:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:26:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:26:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:26:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:26:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:26:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:26:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:26:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:26:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:26:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:26:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:26:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:26:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:26:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:26:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:26:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:26:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:26:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:26:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:26:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:26:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:26:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:27:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:27:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:27:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:27:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:27:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:27:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:27:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:27:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:27:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:27:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:27:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:27:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:27:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:27:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:27:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:27:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:27:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:27:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:27:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:27:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:27:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:27:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:27:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:27:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:27:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:27:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:27:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:28:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:28:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:28:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:28:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:28:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:28:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:28:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:28:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:28:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:28:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:28:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:28:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:28:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:28:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:28:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:28:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:28:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:28:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:28:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:28:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:28:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:28:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:28:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:28:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:28:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:28:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:28:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:28:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:29:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:29:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:29:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:29:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:29:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:29:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:29:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:29:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:29:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:29:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:29:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:29:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:29:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:29:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:29:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:29:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:29:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:29:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:29:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:29:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:29:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:29:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:29:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:29:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:29:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:29:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:29:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:30:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:30:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:30:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:30:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:30:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:30:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:30:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:30:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:30:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:30:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:30:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:30:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:30:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:30:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:30:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:30:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:30:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:30:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:30:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:30:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:30:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:30:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:30:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:30:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:30:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:30:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:30:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:31:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:31:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:31:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:31:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:31:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:31:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:31:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:31:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:31:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:31:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:31:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:31:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:31:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:31:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:31:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:31:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:31:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:31:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:31:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:31:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:31:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:31:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:31:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:31:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:31:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:31:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:31:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:32:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:32:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:32:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:32:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:32:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:32:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:32:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:32:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:32:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:32:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:32:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:32:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:32:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:32:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:32:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:32:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:32:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:32:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:32:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:32:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:32:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:32:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:32:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:32:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:32:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:32:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:32:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:32:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:33:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:33:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:33:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:33:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:33:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:33:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:33:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:33:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:33:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:33:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:33:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:33:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:33:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:33:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:33:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:33:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:33:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:33:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:33:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:33:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:33:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:33:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:33:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:33:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:33:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:33:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:33:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:34:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:34:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:34:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:34:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:34:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:34:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:34:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:34:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:34:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:34:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:34:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:34:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:34:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:34:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:34:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:34:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:34:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:34:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:34:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:34:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:34:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:34:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:34:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:34:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:34:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:34:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:34:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:35:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:35:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:35:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:35:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:35:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:35:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:35:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:35:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:35:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:35:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:35:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:35:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:35:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:35:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:35:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:35:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:35:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:35:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:35:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:35:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:35:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:35:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:35:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:35:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:35:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:35:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:35:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:35:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:36:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:36:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:36:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:36:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:36:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:36:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:36:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:36:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:36:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:36:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:36:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:36:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:36:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:36:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:36:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:36:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:36:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:36:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:36:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:36:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:36:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:36:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:36:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:36:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:36:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:36:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:36:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:37:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:37:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:37:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:37:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:37:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:37:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:37:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:37:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:37:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:37:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:37:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:37:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:37:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:37:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:37:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:37:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:37:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:37:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:37:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:37:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:37:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:37:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:37:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:37:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:37:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:37:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:37:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:38:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:38:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:38:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:38:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:38:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:38:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:38:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:38:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:38:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:38:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:38:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:38:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:38:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:38:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:38:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:38:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:38:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:38:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:38:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:38:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:38:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:38:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:38:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:38:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:38:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:38:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:38:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:38:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:39:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:39:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:39:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:39:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:39:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:39:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:39:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:39:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:39:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:39:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:39:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:39:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:39:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:39:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:39:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:39:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:39:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:39:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:39:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:39:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:39:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:39:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:39:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:39:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:39:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:39:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:39:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:40:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:40:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:40:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:40:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:40:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:40:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:40:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:40:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:40:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:40:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:40:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:40:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:40:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:40:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:40:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:40:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:40:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:40:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:40:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:40:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:40:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:40:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:40:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:40:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:40:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:40:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:40:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:41:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:41:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:41:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:41:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:41:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:41:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:41:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:41:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:41:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:41:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:41:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:41:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:41:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:41:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:41:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:41:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:41:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:41:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:41:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:41:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:41:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:41:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:41:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:41:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:41:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:41:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:41:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:41:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:42:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:42:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:42:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:42:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:42:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:42:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:42:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:42:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:42:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:42:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:42:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:42:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:42:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:42:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:42:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:42:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:42:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:42:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:42:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:42:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:42:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:42:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:42:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:42:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:42:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:42:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:42:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:43:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:43:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:43:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:43:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:43:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:43:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:43:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:43:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:43:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:43:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:43:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:43:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:43:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:43:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:43:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:43:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:43:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:43:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:43:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:43:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:43:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:43:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:43:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:43:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:43:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:43:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:43:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:44:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:44:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:44:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:44:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:44:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:44:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:44:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:44:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:44:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:44:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:44:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:44:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:44:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:44:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:44:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:44:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:44:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:44:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:44:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:44:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:44:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:44:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:44:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:44:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:44:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:44:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:44:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:44:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:45:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:45:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:45:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:45:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:45:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:45:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:45:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:45:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:45:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:45:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:45:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:45:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:45:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:45:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:45:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:45:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:45:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:45:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:45:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:45:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:45:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:45:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:45:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:45:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:45:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:45:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:45:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:46:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:46:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:46:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:46:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:46:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:46:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:46:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:46:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:46:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:46:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:46:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:46:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:46:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:46:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:46:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:46:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:46:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:46:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:46:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:46:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:46:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:46:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:46:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:46:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:46:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:46:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:46:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:47:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:47:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:47:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:47:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:47:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:47:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:47:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:47:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:47:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:47:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:47:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:47:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:47:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:47:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:47:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:47:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:47:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:47:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:47:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:47:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:47:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:47:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:47:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:47:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:47:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:47:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:47:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:48:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:48:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:48:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:48:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:48:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:48:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:48:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:48:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:48:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:48:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:48:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:48:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:48:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:48:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:48:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:48:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:48:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:48:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:48:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:48:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:48:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:48:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:48:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:48:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:48:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:48:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:48:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:48:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:49:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:49:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:49:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:49:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:49:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:49:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:49:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:49:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:49:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:49:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:49:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:49:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:49:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:49:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:49:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:49:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:49:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:49:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:49:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:49:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:49:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:49:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:49:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:49:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:49:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:49:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:49:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:50:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:50:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:50:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:50:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:50:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:50:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:50:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:50:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:50:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:50:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:50:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:50:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:50:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:50:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:50:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:50:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:50:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:50:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:50:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:50:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:50:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:50:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:50:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:50:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:50:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:50:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:50:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:51:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:51:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:51:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:51:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:51:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:51:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:51:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:51:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:51:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:51:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:51:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:51:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:51:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:51:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:51:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:51:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:51:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:51:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:51:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:51:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:51:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:51:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:51:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:51:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:51:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:51:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:51:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:51:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:52:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:52:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:52:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:52:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:52:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:52:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:52:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:52:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:52:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:52:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:52:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:52:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:52:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:52:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:52:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:52:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:52:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:52:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:52:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:52:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:52:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:52:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:52:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:52:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:52:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:52:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:52:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:53:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:53:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:53:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:53:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:53:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:53:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:53:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:53:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:53:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:53:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:53:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:53:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:53:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:53:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:53:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:53:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:53:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:53:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:53:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:53:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:53:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:53:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:53:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:53:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:53:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:53:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:53:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:54:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:54:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:54:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:54:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:54:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:54:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:54:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:54:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:54:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:54:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:54:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:54:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:54:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:54:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:54:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:54:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:54:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:54:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:54:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:54:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:54:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:54:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:54:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:54:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:54:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:54:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:54:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:54:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:55:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:55:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:55:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:55:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:55:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:55:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:55:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:55:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:55:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:55:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:55:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:55:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:55:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:55:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:55:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:55:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:55:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:55:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:55:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:55:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:55:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:55:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:55:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:55:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:55:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:55:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:55:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:56:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:56:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:56:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:56:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:56:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:56:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:56:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:56:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:56:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:56:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:56:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:56:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:56:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:56:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:56:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:56:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:56:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:56:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:56:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:56:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:56:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:56:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:56:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:56:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:56:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:56:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:56:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:57:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:57:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:57:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:57:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:57:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:57:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:57:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:57:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:57:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:57:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:57:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:57:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:57:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:57:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:57:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:57:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:57:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:57:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:57:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:57:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:57:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:57:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:57:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:57:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:57:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:57:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:57:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:57:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:58:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:58:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:58:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:58:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:58:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:58:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:58:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:58:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:58:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:58:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:58:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:58:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:58:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:58:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:58:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:58:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:58:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:58:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:58:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:58:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:58:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:58:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:58:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:58:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:58:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:58:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:58:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:59:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:59:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:59:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:59:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:59:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:59:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:59:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:59:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:59:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:59:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:59:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:59:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:59:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:59:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:59:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:59:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:59:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:59:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:59:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:59:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:59:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:59:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 19:59:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 19:59:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 19:59:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 19:59:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 19:59:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:00:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:00:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:00:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:00:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:00:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:00:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:00:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:00:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:00:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:00:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:00:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:00:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:00:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:00:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:00:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:00:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:00:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:00:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:00:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:00:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:00:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:00:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:00:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:00:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:00:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:01:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:01:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:01:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:01:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:01:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:01:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:01:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:01:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:01:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:01:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:01:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:01:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:01:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:01:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:01:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:01:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:01:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:01:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:01:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:01:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:01:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:01:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:01:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:01:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:01:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:01:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:01:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:02:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:02:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:02:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:02:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:02:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:02:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:02:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:02:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:02:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:02:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:02:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:02:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:02:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:02:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:02:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:02:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:02:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:02:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:02:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:02:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:02:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:02:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:02:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:02:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:02:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:02:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:02:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:03:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:03:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:03:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:03:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:03:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:03:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:03:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:03:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:03:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:03:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:03:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:03:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:03:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:03:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:03:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:03:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:03:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:03:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:03:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:03:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:03:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:03:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:03:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:03:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:03:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:03:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:03:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:04:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:04:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:04:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:04:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:04:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:04:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:04:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:04:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:04:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:04:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:04:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:04:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:04:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:04:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:04:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:04:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:04:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:04:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:04:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:04:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:04:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:04:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:04:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:04:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:04:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:04:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:04:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:04:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:05:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:05:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:05:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:05:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:05:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:05:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:05:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:05:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:05:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:05:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:05:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:05:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:05:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:05:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:05:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:05:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:05:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:05:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:05:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:05:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:05:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:05:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:05:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:05:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:05:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:05:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:05:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:06:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:06:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:06:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:06:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:06:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:06:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:06:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:06:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:06:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:06:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:06:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:06:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:06:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:06:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:06:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:06:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:06:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:06:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:06:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:06:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:06:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:06:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:06:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:06:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:07:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:07:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:07:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:07:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:07:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:07:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:07:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:07:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:07:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:07:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:07:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:07:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:07:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:07:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:07:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:07:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:07:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:07:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:07:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:07:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:07:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:07:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:07:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:07:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:07:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:08:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:08:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:08:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:08:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:08:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:08:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:08:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:08:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:08:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:08:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:08:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:08:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:08:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:08:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:08:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:08:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:08:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:08:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:08:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:08:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:08:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:08:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:08:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:08:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:08:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:08:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:08:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:09:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:09:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:09:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:09:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:09:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:09:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:09:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:09:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:09:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:09:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:09:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:09:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:09:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:09:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:09:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:09:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:09:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:09:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:09:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:09:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:09:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:09:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:09:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:09:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:09:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:09:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:09:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:10:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:10:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:10:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:10:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:10:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:10:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:10:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:10:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:10:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:10:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:10:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:10:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:10:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:10:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:10:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:10:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:10:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:10:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:10:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:10:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:10:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:10:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:10:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:10:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:10:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:10:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:10:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:10:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:11:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:11:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:11:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:11:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:11:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:11:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:11:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:11:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:11:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:11:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:11:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:11:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:11:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:11:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:11:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:11:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:11:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:11:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:11:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:11:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:11:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:11:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:11:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:11:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:11:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:11:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:11:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:12:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:12:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:12:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:12:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:12:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:12:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:12:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:12:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:12:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:12:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:12:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:12:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:12:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:12:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:12:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:12:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:12:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:12:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:12:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:12:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:12:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:12:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:12:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:12:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:13:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:13:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:13:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:13:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:13:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:13:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:13:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:13:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:13:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:13:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:13:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:13:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:13:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:13:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:13:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:13:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:13:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:13:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:13:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:13:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:13:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:13:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:13:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:13:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:13:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:13:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:13:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:13:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:14:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:14:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:14:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:14:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:14:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:14:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:14:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:14:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:14:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:14:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:14:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:14:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:14:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:14:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:14:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:14:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:14:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:14:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:14:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:14:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:14:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:14:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:14:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:14:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:14:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:14:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:14:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:15:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:15:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:15:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:15:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:15:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:15:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:15:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:15:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:15:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:15:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:15:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:15:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:15:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:15:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:15:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:15:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:15:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:15:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:15:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:15:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:15:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:15:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:15:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:15:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:16:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:16:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:16:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:16:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:16:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:16:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:16:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:16:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:16:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:16:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:16:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:16:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:16:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:16:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:16:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:16:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:16:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:16:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:16:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:16:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:16:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:16:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:16:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:16:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:16:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:16:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:16:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:16:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:17:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:17:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:17:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:17:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:17:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:17:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:17:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:17:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:17:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:17:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:17:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:17:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:17:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:17:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:17:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:17:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:17:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:17:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:17:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:17:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:17:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:17:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:17:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:17:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:17:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:17:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:17:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:18:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:18:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:18:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:18:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:18:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:18:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:18:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:18:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:18:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:18:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:18:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:18:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:18:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:18:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:18:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:18:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:18:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:18:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:18:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:18:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:18:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:18:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:18:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:18:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:18:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:18:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:18:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:19:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:19:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:19:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:19:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:19:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:19:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:19:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:19:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:19:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:19:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:19:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:19:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:19:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:19:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:19:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:19:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:19:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:19:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:19:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:19:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:19:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:19:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:19:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:19:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:19:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:19:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:19:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:20:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:20:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:20:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:20:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:20:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:20:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:20:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:20:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:20:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:20:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:20:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:20:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:20:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:20:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:20:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:20:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:20:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:20:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:20:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:20:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:20:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:20:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:20:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:20:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:20:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:21:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:21:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:21:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:21:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:21:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:21:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:21:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:21:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:21:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:21:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:21:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:21:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:21:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:21:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:21:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:21:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:21:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:21:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:21:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:21:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:21:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:21:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:21:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:21:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:21:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:21:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:21:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:22:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:22:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:22:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:22:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:22:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:22:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:22:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:22:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:22:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:22:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:22:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:22:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:22:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:22:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:22:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:22:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:22:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:22:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:22:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:22:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:22:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:22:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:22:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:22:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:23:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:23:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:23:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:23:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:23:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:23:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:23:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:23:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:23:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:23:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:23:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:23:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:23:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:23:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:23:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:23:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:23:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:23:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:23:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:23:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:23:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:23:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:24:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:24:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:24:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:24:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:24:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:24:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:24:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:24:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:24:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:24:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:24:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:24:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:24:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:24:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:24:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:24:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:24:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:24:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:24:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:24:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:24:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:24:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:24:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:24:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:25:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:25:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:25:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:25:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:25:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:25:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:25:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:25:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:25:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:25:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:25:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:25:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:25:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:25:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:25:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:25:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:25:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:25:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:25:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:25:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:25:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:25:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:25:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:25:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:26:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:26:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:26:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:26:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:26:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:26:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:26:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:26:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:26:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:26:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:26:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:26:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:26:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:26:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:26:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:26:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:26:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:26:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:26:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:26:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:26:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:26:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:26:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:26:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:26:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:26:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:26:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:26:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:27:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:27:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:27:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:27:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:27:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:27:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:27:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:27:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:27:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:27:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:27:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:27:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:27:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:27:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:27:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:27:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:27:46 rdma06 systemd[1]: Starting dnf makecache...
Apr  7 20:27:46 rdma06 dnf[597901]: Updating Subscription Management repositories.
Apr  7 20:27:46 rdma06 dnf[597901]: Unable to read consumer identity
Apr  7 20:27:46 rdma06 dnf[597901]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  7 20:27:47 rdma06 dnf[597901]: Copr repo for fmf owned by psss                 3.0 kB/s | 3.3 kB     00:01
Apr  7 20:27:47 rdma06 dnf[597901]: beaker-AppStream                                 74 kB/s | 3.2 kB     00:00
Apr  7 20:27:47 rdma06 dnf[597901]: beaker-AppStream-debuginfo                       56 kB/s | 1.5 kB     00:00
Apr  7 20:27:48 rdma06 dnf[597901]: beaker-BaseOS                                    73 kB/s | 2.7 kB     00:00
Apr  7 20:27:48 rdma06 dnf[597901]: beaker-BaseOS-debuginfo                          49 kB/s | 1.5 kB     00:00
Apr  7 20:27:48 rdma06 dnf[597901]: beaker-CRB                                       35 kB/s | 2.7 kB     00:00
Apr  7 20:27:48 rdma06 dnf[597901]: beaker-CRB-debuginfo                             66 kB/s | 1.5 kB     00:00
Apr  7 20:27:48 rdma06 dnf[597901]: beaker-HighAvailability                          73 kB/s | 2.8 kB     00:00
Apr  7 20:27:48 rdma06 dnf[597901]: beaker-HighAvailability-debuginfo                38 kB/s | 1.5 kB     00:00
Apr  7 20:27:48 rdma06 dnf[597901]: beaker-NFV                                       58 kB/s | 2.7 kB     00:00
Apr  7 20:27:48 rdma06 dnf[597901]: beaker-NFV-debuginfo                             69 kB/s | 1.5 kB     00:00
Apr  7 20:27:48 rdma06 dnf[597901]: beaker-RT                                        62 kB/s | 2.7 kB     00:00
Apr  7 20:27:48 rdma06 dnf[597901]: beaker-RT-debuginfo                              76 kB/s | 1.5 kB     00:00
Apr  7 20:27:48 rdma06 dnf[597901]: beaker-ResilientStorage                          90 kB/s | 2.8 kB     00:00
Apr  7 20:27:48 rdma06 dnf[597901]: beaker-ResilientStorage-debuginfo                58 kB/s | 1.5 kB     00:00
Apr  7 20:27:48 rdma06 dnf[597901]: beaker-SAP                                       73 kB/s | 2.7 kB     00:00
Apr  7 20:27:48 rdma06 dnf[597901]: beaker-SAP-debuginfo                             59 kB/s | 1.5 kB     00:00
Apr  7 20:27:48 rdma06 dnf[597901]: beaker-SAPHANA                                   88 kB/s | 2.7 kB     00:00
Apr  7 20:27:48 rdma06 dnf[597901]: beaker-SAPHANA-debuginfo                         47 kB/s | 1.5 kB     00:00
Apr  7 20:27:49 rdma06 dnf[597901]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr  7 20:27:50 rdma06 dnf[597901]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr  7 20:27:50 rdma06 dnf[597901]: Metadata cache created.
Apr  7 20:27:50 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  7 20:27:50 rdma06 systemd[1]: Finished dnf makecache.
Apr  7 20:27:50 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.387s CPU time.
Apr  7 20:27:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:27:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:27:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:27:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:27:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:28:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:28:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:28:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:28:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:28:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:28:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:28:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:28:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:28:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:28:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:28:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:28:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:28:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:28:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:28:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:28:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:28:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:28:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:28:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:28:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:28:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:28:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:28:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:28:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:29:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:29:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:29:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:29:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:29:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:29:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:29:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:29:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:29:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:29:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:29:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:29:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:29:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:29:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:29:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:29:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:29:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:29:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:29:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:29:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:29:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:29:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:29:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:29:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:29:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:29:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:29:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:29:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:30:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:30:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:30:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:30:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:30:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:30:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:30:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:30:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:30:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:30:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:30:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:30:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:30:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:30:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:30:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:30:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:30:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:30:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:30:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:30:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:30:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:31:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:31:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:31:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:31:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:31:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:31:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:31:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:31:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:31:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:31:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:31:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:31:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:31:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:31:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:31:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:31:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:31:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:31:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:31:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:31:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:31:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:32:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:32:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:32:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:32:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:32:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:32:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:32:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:32:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:32:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:32:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:32:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:32:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:32:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:32:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:32:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:32:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:32:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:32:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:32:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:32:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:32:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:32:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:32:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:32:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:32:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:32:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:32:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:32:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:33:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:33:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:33:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:33:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:33:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:33:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:33:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:33:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:33:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:33:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:33:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:33:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:33:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:33:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:33:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:33:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:33:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:33:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:33:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:33:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:33:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:33:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:33:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:33:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:34:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:34:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:34:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:34:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:34:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:34:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:34:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:34:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:34:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:34:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:34:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:34:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:34:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:34:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:34:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:34:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:34:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:34:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:35:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:35:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:35:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:35:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:35:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:35:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:35:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:35:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:35:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:35:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:35:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:35:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:35:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:35:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:35:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:35:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:35:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:35:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:35:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:35:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:35:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:35:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:35:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:35:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:35:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:36:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:36:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:36:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:36:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:36:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:36:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:36:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:36:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:36:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:36:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:36:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:36:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:36:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:36:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:36:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:36:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:36:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:36:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:36:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:36:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:36:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:36:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:36:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:36:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:36:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:36:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:36:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:37:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:37:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:37:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:37:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:37:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:37:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:37:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:37:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:37:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:37:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:37:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:37:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:37:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:37:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:37:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:37:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:37:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:37:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:37:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:37:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:37:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:37:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:37:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:37:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:37:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:37:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:37:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:38:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:38:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:38:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:38:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:38:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:38:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:38:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:38:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:38:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:38:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:38:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:38:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:38:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:38:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:38:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:38:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:38:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:38:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:38:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:38:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:38:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:38:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:38:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:38:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:39:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:39:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:39:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:39:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:39:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:39:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:39:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:39:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:39:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:39:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:39:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:39:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:39:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:39:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:39:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:39:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:39:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:39:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:39:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:39:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:39:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:39:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:40:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:40:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:40:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:40:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:40:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:40:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:40:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:40:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:40:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:40:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:40:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:40:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:40:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:40:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:40:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:40:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:40:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:40:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:40:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:40:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:40:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:40:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:40:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:40:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:40:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:40:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:40:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:41:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:41:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:41:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:41:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:41:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:41:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:41:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:41:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:41:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:41:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:41:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:41:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:41:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:41:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:41:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:41:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:41:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:41:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:41:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:41:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:41:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:41:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:41:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:41:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:42:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:42:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:42:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:42:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:42:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:42:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:42:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:42:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:42:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:42:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:42:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:42:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:42:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:42:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:42:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:42:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:42:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:42:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:42:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:42:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:42:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:42:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:42:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:42:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:42:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:43:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:43:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:43:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:43:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:43:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:43:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:43:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:43:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:43:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:43:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:43:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:43:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:43:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:43:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:43:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:44:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:44:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:44:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:44:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:44:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:44:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:44:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:44:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:44:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:44:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:44:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:44:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:44:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:44:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:44:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:44:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:44:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:44:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:44:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:44:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:44:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:45:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:45:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:45:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:45:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:45:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:45:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:45:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:45:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:45:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:45:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:45:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:45:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:45:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:45:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:45:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:45:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:45:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:45:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:45:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:45:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:45:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:45:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:46:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:46:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:46:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:46:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:46:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:46:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:46:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:46:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:46:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:46:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:46:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:46:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:46:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:46:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:46:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:46:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:46:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:46:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:46:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:46:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:46:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:46:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:46:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:46:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:47:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:47:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:47:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:47:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:47:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:47:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:47:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:47:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:47:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:47:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:47:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:47:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:47:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:47:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:47:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:47:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:47:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:47:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:48:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:48:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:48:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:48:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:48:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:48:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:48:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:48:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:48:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:48:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:48:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:48:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:48:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:48:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:48:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:48:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:48:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:48:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:48:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:48:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:48:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:48:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:49:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:49:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:49:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:49:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:49:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:49:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:49:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:49:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:49:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:49:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:49:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:49:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:49:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:49:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:49:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:50:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:50:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:50:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:50:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:50:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:50:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:50:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:50:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:50:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:50:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:50:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:50:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:50:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:50:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:50:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:50:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:50:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:50:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:51:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:51:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:51:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:51:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:51:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:51:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:51:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:51:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:51:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:51:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:51:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:51:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:51:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:51:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:51:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:51:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:52:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:52:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:52:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:52:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:52:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:52:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:52:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:52:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:52:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:52:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:52:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:52:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:52:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:52:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:52:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:52:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:52:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:52:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:52:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:52:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:52:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:53:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:53:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:53:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:53:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:53:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:53:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:53:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:53:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:53:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:53:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:53:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:53:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:54:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:54:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:54:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:54:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:54:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:54:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:54:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:54:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:54:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:54:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:54:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:54:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:54:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:54:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:54:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:55:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:55:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:55:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:55:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:55:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:55:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:55:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:55:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:55:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:55:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:55:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:55:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:55:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:55:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:55:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:55:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:55:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:55:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:55:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:56:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:56:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:56:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:56:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:56:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:56:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:56:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:56:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:56:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:57:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:57:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:57:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:57:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:57:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:57:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:57:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:57:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:57:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:57:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:57:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:57:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:57:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:57:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:57:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:57:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:57:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:57:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:58:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:58:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:58:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:58:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:58:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:58:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:58:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:58:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:58:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:58:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:59:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:59:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:59:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:59:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:59:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:59:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:59:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:59:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:59:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:59:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:59:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:59:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:59:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 20:59:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 20:59:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 20:59:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 20:59:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 20:59:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:00:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:00:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:00:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:00:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:00:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:00:35 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:00:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:00:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:00:35 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:00:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:00:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:00:55 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:00:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:00:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:00:55 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:01:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:01:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:01:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:01:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:01:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:01:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:01:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:01:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:01:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:01:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:01:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:01:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:01:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:01:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:01:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:01:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:02:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:02:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:02:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:02:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:02:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:02:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:02:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:02:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:02:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:02:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:02:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:02:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:03:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:03:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:03:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:03:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:03:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:03:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:03:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:03:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:03:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:03:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:03:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:03:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:04:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:04:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:04:15 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:04:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:04:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:04:15 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:04:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:04:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:04:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:04:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:04:45 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:04:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:04:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:04:45 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:04:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:04:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:05:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:05:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:05:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:05:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:05:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:05:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:05:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:05:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:05:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:06:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:06:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:06:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:06:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:06:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:06:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:06:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:06:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:06:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:07:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:07:05 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:07:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:07:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:07:05 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:07:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:07:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:07:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:07:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:07:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:07:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:07:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:07:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:08:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:08:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:08:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:08:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:08:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:08:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:08:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:08:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:08:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:08:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:08:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:08:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:09:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:09:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:09:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:09:25 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:09:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:09:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:09:25 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:09:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:09:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:09:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:09:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:09:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:10:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:10:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:10:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:10:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:10:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:10:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:10:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:10:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:10:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:11:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:11:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:11:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:11:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:11:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:11:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:11:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:11:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:11:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:11:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:12:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:12:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:12:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:12:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:12:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:12:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:12:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:12:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:12:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:13:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:13:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:13:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:13:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:13:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:13:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:13:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:13:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:13:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:14:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:14:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:14:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:14:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:14:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:14:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:14:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:14:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:14:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:14:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:15:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:15:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:15:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:15:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:15:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:15:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:15:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:15:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:15:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:16:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:16:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:16:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:16:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:16:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:16:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:16:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:16:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:16:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:17:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:17:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:17:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:17:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:17:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:17:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:17:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:17:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:17:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:17:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:18:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:18:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:18:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:18:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:18:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:18:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:18:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:18:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:18:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:19:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:19:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:19:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:19:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:19:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:19:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:19:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:19:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:19:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:20:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:20:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:20:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:20:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:20:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:20:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:20:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:20:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:20:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:20:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:21:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:21:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:21:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:21:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:21:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:21:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:21:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:21:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:21:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:22:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:22:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:22:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:22:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:22:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:22:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:22:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:22:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:22:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:23:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:23:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:23:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:23:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:23:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:23:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:23:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:23:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:23:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:23:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:24:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:24:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:24:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:24:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:24:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:24:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:24:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:24:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:24:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:25:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:25:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:25:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:25:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:25:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:25:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:25:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:25:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:25:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:26:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:26:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:26:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:26:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:26:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:26:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:26:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:26:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:26:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:27:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:27:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:27:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:27:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:27:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:27:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:27:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:27:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:27:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:27:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:28:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:28:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:28:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:28:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:28:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:28:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:28:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:28:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:28:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:29:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:29:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:29:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:29:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:29:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:29:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:29:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:29:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:29:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:30:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:30:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:30:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:30:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:30:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:30:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:30:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:30:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:30:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:30:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:31:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:31:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:31:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:31:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:31:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:31:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:31:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:31:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:31:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:32:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:32:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:32:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:32:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:32:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:32:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:32:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:32:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:32:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:33:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:33:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:33:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:33:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:33:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:33:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:33:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:33:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:33:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:33:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:34:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:34:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:34:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:34:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:34:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:34:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:34:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:34:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:34:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:35:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:35:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:35:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:35:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:35:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:35:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:35:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:35:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:35:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:36:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:36:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:36:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:36:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:36:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:36:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:36:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:36:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:36:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:36:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:37:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:37:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:37:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:37:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:37:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:37:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:37:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:37:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:37:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:38:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:38:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:38:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:38:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:38:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:38:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:38:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:38:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:38:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:39:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:39:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:39:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:39:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:39:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:39:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:39:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:39:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:39:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:39:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:40:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:40:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:40:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:40:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:40:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:40:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:40:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:40:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:40:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:41:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:41:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:41:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:41:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:41:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:41:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:41:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:41:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:41:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:42:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:42:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:42:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:42:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:42:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:42:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:42:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:42:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:42:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:42:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:43:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:43:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:43:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:43:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:43:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:43:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:43:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:43:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:43:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:44:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:44:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:44:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:44:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:44:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:44:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:44:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:44:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:44:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:45:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:45:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:45:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:45:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:45:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:45:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:45:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:45:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:45:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:46:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:46:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:46:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:46:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:46:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:46:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:46:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:46:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:46:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:46:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:47:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:47:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:47:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:47:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:47:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:47:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:47:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:47:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:47:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:48:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:48:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:48:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:48:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:48:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:48:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:48:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:48:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:48:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:49:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:49:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:49:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:49:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:49:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:49:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:49:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:49:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:49:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:49:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:50:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:50:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:50:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:50:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:50:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:50:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:50:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:50:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:50:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:51:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:51:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:51:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:51:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:51:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:51:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:51:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:51:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:51:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:52:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:52:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:52:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:52:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:52:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:52:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:52:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:52:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:52:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:52:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:53:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:53:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:53:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:53:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:53:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:53:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:53:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:53:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:53:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:54:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:54:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:54:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:54:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:54:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:54:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:54:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:54:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:54:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:55:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:55:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:55:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:55:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:55:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:55:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:55:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:55:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:55:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:55:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:56:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:56:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:56:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:56:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:56:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:56:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:56:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:56:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:56:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:57:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:57:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:57:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:57:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:57:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:57:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:57:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:57:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:57:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:58:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:58:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:58:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:58:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:58:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:58:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:58:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:58:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:58:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:58:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:59:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:59:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:59:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:59:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:59:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:59:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:59:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 21:59:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 21:59:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 21:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 21:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 21:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:00:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:00:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:00:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:00:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:00:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:00:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:00:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:00:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:00:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:01:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:01:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:01:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:01:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:01:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:01:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:01:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:01:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:01:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:02:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:02:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:02:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:02:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:02:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:02:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:02:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:02:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:02:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:02:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:03:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:03:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:03:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:03:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:03:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:03:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:03:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:03:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:03:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:04:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:04:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:04:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:04:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:04:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:04:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:04:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:04:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:04:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:05:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:05:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:05:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:05:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:05:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:05:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:05:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:05:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:05:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:05:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:06:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:06:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:06:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:06:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:06:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:06:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:06:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:06:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:06:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:07:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:07:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:07:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:07:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:07:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:07:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:07:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:07:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:07:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:08:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:08:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:08:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:08:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:08:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:08:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:08:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:08:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:08:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:08:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:09:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:09:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:09:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:09:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:09:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:09:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:09:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:09:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:09:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:10:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:10:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:10:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:10:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:10:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:10:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:10:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:10:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:10:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:11:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:11:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:11:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:11:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:11:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:11:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:11:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:11:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:11:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:11:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:12:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:12:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:12:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:12:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:12:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:12:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:12:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:12:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:12:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:13:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:13:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:13:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:13:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:13:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:13:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:13:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:13:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:13:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:14:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:14:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:14:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:14:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:14:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:14:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:14:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:14:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:14:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:14:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:15:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:15:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:15:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:15:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:15:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:15:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:15:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:15:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:15:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:16:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:16:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:16:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:16:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:16:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:16:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:16:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:16:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:16:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:17:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:17:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:17:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:17:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:17:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:17:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:17:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:17:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:17:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:18:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:18:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:18:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:18:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:18:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:18:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:18:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:18:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:18:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:18:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:19:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:19:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:19:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:19:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:19:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:19:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:19:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:19:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:19:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:20:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:20:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:20:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:20:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:20:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:20:33 rdma06 NetworkManager[1025]: <info>  [1680920433.0426] dhcp4 (eno1): state changed new lease, address=10.73.131.45
Apr  7 22:20:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:20:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:20:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:20:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:21:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:21:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:21:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:21:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:21:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:21:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:21:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:21:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:21:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:21:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:22:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:22:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:22:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:22:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:22:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:22:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:22:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:22:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:22:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:23:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:23:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:23:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:23:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:23:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:23:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:23:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:23:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:23:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:24:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:24:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:24:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:24:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:24:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:24:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:24:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:24:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:24:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:24:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:25:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  7 22:25:01 rdma06 dnf[602656]: Updating Subscription Management repositories.
Apr  7 22:25:01 rdma06 dnf[602656]: Unable to read consumer identity
Apr  7 22:25:01 rdma06 dnf[602656]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  7 22:25:01 rdma06 dnf[602656]: Metadata cache refreshed recently.
Apr  7 22:25:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  7 22:25:01 rdma06 systemd[1]: Finished dnf makecache.
Apr  7 22:25:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:25:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:25:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:25:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:25:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:25:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:25:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:25:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:25:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:26:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:26:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:26:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:26:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:26:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:26:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:26:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:26:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:26:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:27:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:27:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:27:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:27:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:27:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:27:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:27:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:27:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:27:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:27:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:28:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:28:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:28:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:28:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:28:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:28:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:28:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:28:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:28:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:29:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:29:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:29:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:29:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:29:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:29:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:29:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:29:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:29:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:30:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:30:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:30:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:30:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:30:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:30:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:30:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:30:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:30:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:30:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:31:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:31:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:31:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:31:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:31:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:31:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:31:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:31:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:31:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:32:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:32:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:32:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:32:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:32:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:32:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:32:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:32:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:32:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:33:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:33:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:33:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:33:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:33:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:33:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:33:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:33:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:33:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:34:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:34:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:34:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:34:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:34:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:34:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:34:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:34:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:34:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:34:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:35:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:35:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:35:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:35:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:35:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:35:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:35:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:35:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:35:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:36:00 rdma06 systemd[5685]: Starting Cleanup of User's Temporary Files and Directories...
Apr  7 22:36:00 rdma06 systemd[5685]: Finished Cleanup of User's Temporary Files and Directories.
Apr  7 22:36:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:36:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:36:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:36:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:36:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:36:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:36:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:36:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:36:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:37:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:37:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:37:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:37:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:37:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:37:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:37:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:37:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:37:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:37:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:38:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:38:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:38:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:38:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:38:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:38:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:38:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:38:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:38:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:39:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:39:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:39:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:39:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:39:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:39:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:39:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:39:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:39:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:40:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:40:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:40:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:40:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:40:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:40:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:40:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:40:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:40:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:40:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:41:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:41:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:41:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:41:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:41:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:41:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:41:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:41:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:41:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:42:00 rdma06 systemd[1]: Starting Cleanup of Temporary Directories...
Apr  7 22:42:00 rdma06 systemd[1]: systemd-tmpfiles-clean.service: Deactivated successfully.
Apr  7 22:42:00 rdma06 systemd[1]: Finished Cleanup of Temporary Directories.
Apr  7 22:42:00 rdma06 systemd[1]: run-credentials-systemd\x2dtmpfiles\x2dclean.service.mount: Deactivated successfully.
Apr  7 22:42:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:42:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:42:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:42:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:42:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:42:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:42:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:42:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:42:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:43:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:43:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:43:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:43:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:43:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:43:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:43:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:43:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:43:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:43:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:44:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:44:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:44:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:44:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:44:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:44:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:44:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:44:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:44:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:45:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:45:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:45:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:45:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:45:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:45:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:45:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:45:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:45:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:46:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:46:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:46:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:46:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:46:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:46:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:46:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:46:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:46:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:46:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:47:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:47:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:47:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:47:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:47:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:47:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:47:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:47:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:47:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:48:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:48:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:48:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:48:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:48:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:48:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:48:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:48:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:48:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:49:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:49:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:49:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:49:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:49:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:49:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:49:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:49:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:49:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:49:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:50:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:50:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:50:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:50:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:50:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:50:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:50:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:50:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:50:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:51:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:51:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:51:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:51:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:51:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:51:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:51:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:51:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:51:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:52:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:52:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:52:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:52:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:52:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:52:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:52:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:52:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:52:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:53:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:53:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:53:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:53:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:53:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:53:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:53:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:53:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:53:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:53:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:54:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:54:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:54:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:54:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:54:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:54:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:54:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:54:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:54:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:55:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:55:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:55:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:55:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:55:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:55:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:55:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:55:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:55:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:56:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:56:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:56:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:56:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:56:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:56:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:56:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:56:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:56:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:56:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:57:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:57:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:57:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:57:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:57:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:57:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:57:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:57:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:57:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:58:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:58:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:58:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:58:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:58:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:58:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:58:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:58:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:58:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:59:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:59:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:59:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:59:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:59:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:59:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:59:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 22:59:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:59:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 22:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 22:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 22:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 22:59:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:00:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:00:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:00:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:00:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:00:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:00:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:00:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:00:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:00:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:01:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:01:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:01:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:01:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:01:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:01:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:01:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:01:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:01:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:02:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:02:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:02:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:02:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:02:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:02:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:02:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:02:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:02:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:02:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:03:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:03:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:03:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:03:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:03:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:03:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:03:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:03:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:03:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:04:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:04:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:04:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:04:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:04:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:04:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:04:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:04:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:04:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:05:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:05:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:05:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:05:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:05:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:05:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:05:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:05:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:05:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:05:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:06:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:06:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:06:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:06:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:06:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:06:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:06:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:06:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:06:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:07:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:07:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:07:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:07:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:07:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:07:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:07:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:07:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:07:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:08:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:08:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:08:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:08:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:08:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:08:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:08:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:08:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:08:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:09:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:09:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:09:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:09:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:09:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:09:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:09:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:09:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:09:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:09:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:10:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:10:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:10:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:10:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:10:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:10:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:10:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:10:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:10:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:11:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:11:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:11:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:11:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:11:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:11:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:11:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:11:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:11:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:12:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:12:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:12:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:12:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:12:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:12:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:12:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:12:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:12:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:12:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:13:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:13:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:13:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:13:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:13:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:13:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:13:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:13:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:13:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:14:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:14:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:14:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:14:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:14:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:14:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:14:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:14:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:14:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:15:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:15:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:15:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:15:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:15:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:15:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:15:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:15:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:15:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:15:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:16:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:16:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:16:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:16:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:16:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:16:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:16:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:16:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:16:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:17:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:17:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:17:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:17:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:17:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:17:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:17:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:17:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:17:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:18:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:18:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:18:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:18:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:18:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:18:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:18:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:18:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:18:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:18:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:19:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:19:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:19:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:19:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:19:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:19:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:19:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:19:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:19:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:20:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:20:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:20:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:20:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:20:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:20:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:20:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:20:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:20:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:21:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:21:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:21:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:21:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:21:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:21:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:21:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:21:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:21:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:21:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:22:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:22:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:22:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:22:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:22:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:22:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:22:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:22:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:22:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:23:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:23:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:23:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:23:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:23:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:23:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:23:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:23:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:23:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:24:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:24:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:24:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:24:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:24:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:24:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:24:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:24:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:24:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:25:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:25:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:25:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:25:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:25:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:25:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:25:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:25:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:25:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:25:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:26:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  7 23:26:01 rdma06 dnf[605127]: Updating Subscription Management repositories.
Apr  7 23:26:01 rdma06 dnf[605127]: Unable to read consumer identity
Apr  7 23:26:01 rdma06 dnf[605127]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  7 23:26:01 rdma06 dnf[605127]: Metadata cache refreshed recently.
Apr  7 23:26:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  7 23:26:01 rdma06 systemd[1]: Finished dnf makecache.
Apr  7 23:26:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:26:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:26:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:26:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:26:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:26:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:26:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:26:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:26:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:27:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:27:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:27:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:27:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:27:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:27:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:27:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:27:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:27:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:28:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:28:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:28:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:28:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:28:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:28:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:28:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:28:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:28:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:28:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:29:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:29:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:29:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:29:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:29:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:29:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:29:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:29:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:29:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:30:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:30:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:30:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:30:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:30:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:30:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:30:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:30:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:30:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:31:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:31:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:31:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:31:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:31:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:31:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:31:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:31:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:31:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:31:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:32:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:32:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:32:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:32:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:32:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:32:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:32:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:32:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:32:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:33:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:33:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:33:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:33:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:33:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:33:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:33:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:33:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:33:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:34:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:34:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:34:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:34:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:34:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:34:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:34:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:34:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:34:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:34:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:35:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:35:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:35:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:35:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:35:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:35:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:35:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:35:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:35:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:36:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:36:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:36:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:36:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:36:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:36:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:36:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:36:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:36:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:37:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:37:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:37:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:37:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:37:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:37:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:37:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:37:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:37:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:37:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:38:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:38:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:38:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:38:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:38:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:38:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:38:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:38:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:38:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:39:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:39:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:39:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:39:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:39:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:39:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:39:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:39:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:39:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:40:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:40:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:40:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:40:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:40:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:40:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:40:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:40:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:40:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:41:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:41:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:41:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:41:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:41:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:41:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:41:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:41:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:41:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:41:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:42:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:42:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:42:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:42:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:42:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:42:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:42:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:42:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:42:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:43:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:43:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:43:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:43:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:43:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:43:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:43:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:43:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:43:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:44:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:44:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:44:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:44:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:44:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:44:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:44:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:44:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:44:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:44:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:45:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:45:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:45:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:45:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:45:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:45:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:45:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:45:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:45:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:46:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:46:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:46:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:46:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:46:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:46:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:46:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:46:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:46:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:47:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:47:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:47:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:47:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:47:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:47:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:47:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:47:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:47:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:47:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:48:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:48:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:48:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:48:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:48:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:48:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:48:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:48:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:48:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:49:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:49:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:49:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:49:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:49:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:49:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:49:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:49:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:49:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:50:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:50:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:50:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:50:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:50:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:50:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:50:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:50:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:50:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:50:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:51:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:51:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:51:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:51:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:51:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:51:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:51:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:51:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:51:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:52:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:52:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:52:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:52:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:52:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:52:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:52:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:52:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:52:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:53:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:53:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:53:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:53:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:53:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:53:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:53:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:53:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:53:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:53:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:54:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:54:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:54:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:54:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:54:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:54:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:54:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:54:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:54:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:55:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:55:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:55:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:55:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:55:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:55:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:55:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:55:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:55:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:56:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:56:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:56:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:56:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:56:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:56:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:56:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:56:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:56:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:56:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:57:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:57:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:57:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:57:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:57:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:57:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:57:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:57:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:57:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:58:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:58:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:58:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:58:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:58:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:58:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:58:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:58:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:58:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:59:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:59:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:59:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:59:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:59:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:59:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:59:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  7 23:59:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  7 23:59:49 rdma06 chronyd[1034]: Selected source 10.18.100.10
Apr  7 23:59:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  7 23:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  7 23:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  7 23:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:00:00 rdma06 systemd[1]: Starting Rotate log files...
Apr  8 00:00:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:00:00 rdma06 systemd[1]: logrotate.service: Deactivated successfully.
Apr  8 00:00:00 rdma06 systemd[1]: Finished Rotate log files.
Apr  8 00:00:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:00:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:00:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:00:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:00:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:00:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:00:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:00:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:00:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:01:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:01:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:01:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:01:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:01:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:01:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:01:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:01:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:01:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:02:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:02:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:02:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:02:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:02:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:02:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:02:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:02:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:02:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:03:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:03:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:03:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:03:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:03:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:03:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:03:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:03:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:03:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:03:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:04:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:04:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:04:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:04:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:04:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:04:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:04:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:04:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:04:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:05:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:05:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:05:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:05:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:05:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:05:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:05:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:05:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:05:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:06:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:06:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:06:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:06:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:06:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:06:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:06:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:06:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:06:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:06:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:07:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:07:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:07:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:07:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:07:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:07:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:07:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:07:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:07:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:08:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:08:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:08:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:08:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:08:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:08:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:08:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:08:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:08:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:09:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:09:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:09:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:09:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:09:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:09:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:09:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:09:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:09:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:09:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:10:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:10:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:10:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:10:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:10:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:10:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:10:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:10:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:10:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:11:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:11:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:11:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:11:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:11:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:11:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:11:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:11:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:11:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:12:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:12:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:12:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:12:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:12:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:12:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:12:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:12:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:12:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:12:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:13:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:13:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:13:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:13:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:13:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:13:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:13:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:13:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:13:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:14:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:14:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:14:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:14:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:14:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:14:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:14:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:14:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:14:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:15:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:15:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:15:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:15:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:15:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:15:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:15:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:15:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:15:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:16:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:16:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:16:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:16:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:16:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:16:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:16:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:16:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:16:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:16:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:17:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:17:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:17:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:17:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:17:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:17:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:17:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:17:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:17:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:18:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:18:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:18:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:18:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:18:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:18:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:18:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:18:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:18:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:19:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:19:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:19:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:19:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:19:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:19:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:19:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:19:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:19:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:19:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:20:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:20:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:20:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:20:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:20:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:20:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:20:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:20:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:20:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:21:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:21:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:21:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:21:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:21:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:21:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:21:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:21:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:21:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:22:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:22:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:22:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:22:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:22:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:22:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:22:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:22:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:22:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:22:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:23:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:23:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:23:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:23:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:23:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:23:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:23:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:23:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:23:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:24:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:24:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:24:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:24:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:24:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:24:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:24:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:24:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:24:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:25:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:25:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:25:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:25:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:25:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:25:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:25:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:25:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:25:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:25:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:26:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:26:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:26:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:26:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:26:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:26:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:26:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:26:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:26:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:27:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:27:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:27:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:27:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:27:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:27:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:27:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:27:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:27:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:28:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:28:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:28:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:28:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:28:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:28:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:28:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:28:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:28:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:28:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:29:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:29:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:29:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:29:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:29:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:29:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:29:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:29:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:29:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:30:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:30:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:30:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:30:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:30:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:30:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:30:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:30:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:30:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:31:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:31:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:31:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:31:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:31:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:31:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:31:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:31:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:31:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:32:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:32:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:32:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:32:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:32:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:32:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:32:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:32:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:32:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:32:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:33:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:33:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:33:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:33:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:33:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:33:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:33:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:33:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:33:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:34:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:34:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:34:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:34:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:34:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:34:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:34:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:34:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:34:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:35:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:35:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:35:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:35:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:35:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:35:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:35:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:35:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:35:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:35:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:36:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:36:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:36:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:36:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:36:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:36:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:36:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:36:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:36:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:37:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:37:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:37:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:37:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:37:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:37:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:37:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:37:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:37:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:38:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:38:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:38:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:38:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:38:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:38:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:38:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:38:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:38:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:38:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:39:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:39:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:39:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:39:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:39:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:39:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:39:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:39:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:39:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:40:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:40:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:40:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:40:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:40:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:40:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:40:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:40:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:40:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:41:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:41:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:41:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:41:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:41:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:41:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:41:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:41:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:41:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:41:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:42:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:42:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:42:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:42:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:42:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:42:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:42:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:42:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:42:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:43:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:43:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:43:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:43:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:43:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:43:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:43:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:43:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:43:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:44:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:44:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:44:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:44:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:44:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:44:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:44:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:44:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:44:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:44:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:45:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:45:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:45:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:45:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:45:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:45:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:45:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:45:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:45:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:46:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:46:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:46:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:46:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:46:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:46:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:46:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:46:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:46:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:47:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:47:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:47:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:47:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:47:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:47:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:47:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:47:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:47:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:48:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:48:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:48:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:48:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:48:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:48:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:48:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:48:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:48:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:48:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:49:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:49:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:49:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:49:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:49:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:49:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:49:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:49:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:49:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:50:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:50:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:50:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:50:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:50:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:50:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:50:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:50:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:50:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:51:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:51:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:51:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:51:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:51:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:51:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:51:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:51:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:51:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:51:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:52:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:52:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:52:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:52:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:52:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:52:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:52:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:52:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:52:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:53:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:53:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:53:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:53:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:53:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:53:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:53:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:53:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:53:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:54:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:54:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:54:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:54:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:54:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:54:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:54:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:54:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:54:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:54:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:55:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:55:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:55:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:55:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:55:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:55:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:55:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:55:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:55:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:56:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:56:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:56:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:56:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:56:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:56:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:56:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:56:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:56:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:57:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:57:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:57:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:57:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:57:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:57:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:57:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:57:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:57:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:57:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:58:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:58:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:58:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:58:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:58:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:58:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:58:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:58:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:58:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:59:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:59:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:59:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:59:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:59:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:59:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:59:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 00:59:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 00:59:47 rdma06 chronyd[1034]: Selected source 10.11.160.238
Apr  8 00:59:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 00:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 00:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 00:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:00:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:00:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:00:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:00:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:00:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:00:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:00:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:00:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:00:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:00:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:01:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:01:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:01:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:01:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:01:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:01:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:01:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:01:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:01:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:02:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:02:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:02:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:02:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:02:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:02:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:02:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:02:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:02:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:03:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:03:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:03:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:03:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:03:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:03:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:03:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:03:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:03:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:03:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:04:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:04:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:04:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:04:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:04:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:04:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:04:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:04:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:04:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:05:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:05:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:05:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:05:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:05:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:05:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:05:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:05:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:05:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:06:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:06:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:06:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:06:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:06:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:06:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:06:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:06:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:06:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:07:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:07:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:07:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:07:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:07:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:07:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:07:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:07:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:07:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:07:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:08:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:08:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:08:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:08:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:08:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:08:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:08:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:08:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:08:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:09:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:09:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:09:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:09:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:09:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:09:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:09:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:09:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:09:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:10:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:10:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:10:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:10:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:10:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:10:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:10:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:10:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:10:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:10:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:11:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:11:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:11:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:11:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:11:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:11:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:11:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:11:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:11:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:12:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:12:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:12:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:12:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:12:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:12:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:12:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:12:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:12:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:13:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:13:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:13:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:13:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:13:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:13:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:13:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:13:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:13:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:13:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:14:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:14:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:14:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:14:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:14:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:14:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:14:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:14:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:14:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:15:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:15:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:15:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:15:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:15:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:15:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:15:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:15:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:15:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:16:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:16:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:16:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:16:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:16:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:16:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:16:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:16:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:16:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:16:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:17:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:17:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:17:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:17:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:17:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:17:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:17:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:17:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:17:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:18:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:18:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:18:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:18:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:18:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:18:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:18:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:18:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:18:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:19:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:19:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:19:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:19:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:19:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:19:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:19:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:19:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:19:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:19:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:20:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:20:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:20:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:20:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:20:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:20:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:20:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:20:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:20:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:21:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:21:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:21:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:21:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:21:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:21:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:21:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:21:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:21:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:22:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:22:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:22:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:22:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:22:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:22:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:22:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:22:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:22:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:23:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:23:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:23:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:23:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:23:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:23:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:23:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:23:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:23:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:23:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:24:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:24:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:24:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:24:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:24:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:24:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:24:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:24:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:24:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:25:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:25:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:25:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:25:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:25:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:25:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:25:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:25:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:25:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:26:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  8 01:26:01 rdma06 dnf[609994]: Updating Subscription Management repositories.
Apr  8 01:26:01 rdma06 dnf[609994]: Unable to read consumer identity
Apr  8 01:26:01 rdma06 dnf[609994]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  8 01:26:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:26:03 rdma06 dnf[609994]: Copr repo for fmf owned by psss                 1.9 kB/s | 3.3 kB     00:01
Apr  8 01:26:03 rdma06 dnf[609994]: beaker-AppStream                                 73 kB/s | 3.2 kB     00:00
Apr  8 01:26:03 rdma06 dnf[609994]: beaker-AppStream-debuginfo                       45 kB/s | 1.5 kB     00:00
Apr  8 01:26:03 rdma06 dnf[609994]: beaker-BaseOS                                   118 kB/s | 2.7 kB     00:00
Apr  8 01:26:03 rdma06 dnf[609994]: beaker-BaseOS-debuginfo                          43 kB/s | 1.5 kB     00:00
Apr  8 01:26:03 rdma06 dnf[609994]: beaker-CRB                                       52 kB/s | 2.7 kB     00:00
Apr  8 01:26:03 rdma06 dnf[609994]: beaker-CRB-debuginfo                             70 kB/s | 1.5 kB     00:00
Apr  8 01:26:03 rdma06 dnf[609994]: beaker-HighAvailability                          75 kB/s | 2.8 kB     00:00
Apr  8 01:26:03 rdma06 dnf[609994]: beaker-HighAvailability-debuginfo                34 kB/s | 1.5 kB     00:00
Apr  8 01:26:03 rdma06 dnf[609994]: beaker-NFV                                       78 kB/s | 2.7 kB     00:00
Apr  8 01:26:03 rdma06 dnf[609994]: beaker-NFV-debuginfo                             74 kB/s | 1.5 kB     00:00
Apr  8 01:26:03 rdma06 dnf[609994]: beaker-RT                                        64 kB/s | 2.7 kB     00:00
Apr  8 01:26:04 rdma06 dnf[609994]: beaker-RT-debuginfo                              35 kB/s | 1.5 kB     00:00
Apr  8 01:26:04 rdma06 dnf[609994]: beaker-ResilientStorage                          81 kB/s | 2.8 kB     00:00
Apr  8 01:26:04 rdma06 dnf[609994]: beaker-ResilientStorage-debuginfo                46 kB/s | 1.5 kB     00:00
Apr  8 01:26:04 rdma06 dnf[609994]: beaker-SAP                                      159 kB/s | 2.7 kB     00:00
Apr  8 01:26:04 rdma06 dnf[609994]: beaker-SAP-debuginfo                             58 kB/s | 1.5 kB     00:00
Apr  8 01:26:04 rdma06 dnf[609994]: beaker-SAPHANA                                   53 kB/s | 2.7 kB     00:00
Apr  8 01:26:04 rdma06 dnf[609994]: beaker-SAPHANA-debuginfo                         36 kB/s | 1.5 kB     00:00
Apr  8 01:26:04 rdma06 dnf[609994]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr  8 01:26:05 rdma06 dnf[609994]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr  8 01:26:05 rdma06 dnf[609994]: Metadata cache created.
Apr  8 01:26:05 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  8 01:26:05 rdma06 systemd[1]: Finished dnf makecache.
Apr  8 01:26:05 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.373s CPU time.
Apr  8 01:26:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:26:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:26:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:26:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:26:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:26:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:26:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:26:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:26:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:27:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:27:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:27:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:27:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:27:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:27:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:27:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:27:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:27:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:28:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:28:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:28:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:28:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:28:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:28:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:28:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:28:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:28:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:29:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:29:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:29:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:29:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:29:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:29:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:29:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:29:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:29:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:29:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:30:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:30:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:30:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:30:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:30:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:30:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:30:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:30:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:30:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:31:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:31:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:31:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:31:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:31:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:31:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:31:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:31:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:31:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:32:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:32:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:32:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:32:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:32:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:32:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:32:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:32:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:32:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:32:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:33:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:33:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:33:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:33:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:33:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:33:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:33:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:33:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:33:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:34:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:34:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:34:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:34:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:34:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:34:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:34:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:34:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:34:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:35:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:35:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:35:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:35:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:35:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:35:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:35:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:35:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:35:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:35:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:36:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:36:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:36:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:36:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:36:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:36:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:36:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:36:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:36:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:37:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:37:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:37:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:37:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:37:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:37:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:37:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:37:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:37:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:38:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:38:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:38:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:38:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:38:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:38:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:38:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:38:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:38:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:39:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:39:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:39:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:39:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:39:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:39:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:39:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:39:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:39:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:39:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:40:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:40:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:40:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:40:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:40:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:40:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:40:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:40:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:40:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:41:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:41:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:41:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:41:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:41:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:41:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:41:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:41:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:41:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:42:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:42:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:42:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:42:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:42:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:42:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:42:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:42:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:42:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:42:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:43:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:43:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:43:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:43:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:43:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:43:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:43:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:43:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:43:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:44:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:44:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:44:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:44:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:44:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:44:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:44:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:44:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:44:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:45:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:45:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:45:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:45:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:45:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:45:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:45:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:45:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:45:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:45:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:46:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:46:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:46:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:46:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:46:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:46:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:46:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:46:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:46:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:47:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:47:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:47:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:47:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:47:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:47:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:47:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:47:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:47:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:48:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:48:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:48:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:48:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:48:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:48:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:48:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:48:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:48:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:48:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:49:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:49:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:49:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:49:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:49:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:49:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:49:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:49:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:49:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:50:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:50:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:50:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:50:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:50:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:50:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:50:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:50:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:50:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:51:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:51:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:51:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:51:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:51:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:51:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:51:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:51:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:51:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:51:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:52:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:52:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:52:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:52:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:52:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:52:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:52:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:52:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:52:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:53:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:53:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:53:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:53:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:53:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:53:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:53:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:53:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:53:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:54:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:54:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:54:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:54:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:54:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:54:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:54:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:54:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:54:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:55:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:55:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:55:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:55:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:55:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:55:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:55:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:55:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:55:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:55:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:56:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:56:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:56:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:56:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:56:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:56:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:56:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:56:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:56:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:57:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:57:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:57:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:57:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:57:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:57:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:57:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:57:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:57:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:58:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:58:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:58:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:58:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:58:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:58:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:58:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:58:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:58:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:58:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:59:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:59:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:59:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:59:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:59:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:59:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:59:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 01:59:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 01:59:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 01:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 01:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 01:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:00:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:00:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:00:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:00:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:00:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:00:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:00:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:00:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:00:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:01:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:01:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:01:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:01:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:01:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:01:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:01:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:01:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:01:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:01:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:02:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:02:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:02:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:02:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:02:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:02:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:02:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:02:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:02:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:03:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:03:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:03:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:03:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:03:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:03:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:03:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:03:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:03:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:04:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:04:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:04:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:04:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:04:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:04:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:04:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:04:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:04:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:04:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:05:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:05:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:05:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:05:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:05:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:05:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:05:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:05:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:05:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:06:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:06:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:06:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:06:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:06:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:06:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:06:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:06:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:06:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:07:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:07:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:07:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:07:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:07:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:07:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:07:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:07:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:07:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:07:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:08:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:08:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:08:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:08:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:08:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:08:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:08:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:08:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:08:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:09:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:09:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:09:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:09:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:09:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:09:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:09:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:09:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:09:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:10:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:10:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:10:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:10:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:10:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:10:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:10:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:10:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:10:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:10:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:11:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:11:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:11:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:11:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:11:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:11:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:11:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:11:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:11:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:12:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:12:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:12:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:12:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:12:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:12:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:12:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:12:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:12:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:13:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:13:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:13:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:13:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:13:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:13:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:13:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:13:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:13:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:14:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:14:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:14:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:14:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:14:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:14:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:14:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:14:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:14:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:14:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:15:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:15:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:15:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:15:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:15:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:15:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:15:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:15:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:15:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:16:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:16:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:16:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:16:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:16:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:16:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:16:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:16:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:16:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:17:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:17:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:17:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:17:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:17:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:17:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:17:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:17:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:17:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:17:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:18:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:18:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:18:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:18:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:18:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:18:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:18:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:18:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:18:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:19:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:19:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:19:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:19:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:19:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:19:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:19:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:19:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:19:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:20:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:20:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:20:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:20:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:20:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:20:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:20:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:20:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:20:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:20:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:21:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:21:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:21:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:21:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:21:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:21:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:21:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:21:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:21:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:22:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:22:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:22:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:22:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:22:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:22:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:22:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:22:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:22:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:23:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:23:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:23:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:23:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:23:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:23:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:23:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:23:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:23:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:23:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:24:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:24:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:24:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:24:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:24:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:24:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:24:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:24:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:24:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:25:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:25:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:25:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:25:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:25:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:25:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:25:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:25:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:25:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:26:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:26:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:26:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:26:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:26:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:26:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:26:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:26:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:26:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:26:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:27:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:27:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:27:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:27:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:27:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:27:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:27:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:27:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:27:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:28:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:28:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:28:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:28:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:28:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:28:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:28:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:28:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:28:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:29:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:29:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:29:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:29:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:29:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:29:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:29:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:29:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:29:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:30:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:30:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:30:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:30:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:30:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:30:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:30:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:30:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:30:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:30:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:31:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:31:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:31:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:31:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:31:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:31:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:31:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:31:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:31:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:32:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:32:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:32:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:32:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:32:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:32:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:32:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:32:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:32:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:33:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:33:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:33:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:33:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:33:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:33:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:33:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:33:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:33:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:33:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:34:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:34:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:34:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:34:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:34:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:34:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:34:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:34:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:34:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:35:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:35:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:35:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:35:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:35:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:35:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:35:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:35:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:35:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:36:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:36:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:36:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:36:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:36:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:36:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:36:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:36:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:36:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:36:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:37:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:37:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:37:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:37:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:37:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:37:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:37:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:37:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:37:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:38:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:38:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:38:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:38:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:38:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:38:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:38:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:38:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:38:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:39:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:39:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:39:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:39:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:39:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:39:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:39:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:39:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:39:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:39:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:40:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:40:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:40:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:40:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:40:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:40:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:40:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:40:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:40:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:41:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:41:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:41:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:41:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:41:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:41:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:41:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:41:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:41:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:42:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:42:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:42:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:42:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:42:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:42:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:42:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:42:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:42:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:42:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:43:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:43:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:43:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:43:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:43:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:43:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:43:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:43:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:43:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:44:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  8 02:44:01 rdma06 dnf[613175]: Updating Subscription Management repositories.
Apr  8 02:44:01 rdma06 dnf[613175]: Unable to read consumer identity
Apr  8 02:44:01 rdma06 dnf[613175]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  8 02:44:01 rdma06 dnf[613175]: Metadata cache refreshed recently.
Apr  8 02:44:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  8 02:44:01 rdma06 systemd[1]: Finished dnf makecache.
Apr  8 02:44:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:44:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:44:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:44:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:44:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:44:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:44:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:44:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:44:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:45:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:45:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:45:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:45:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:45:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:45:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:45:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:45:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:45:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:46:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:46:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:46:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:46:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:46:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:46:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:46:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:46:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:46:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:46:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:47:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:47:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:47:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:47:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:47:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:47:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:47:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:47:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:47:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:48:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:48:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:48:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:48:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:48:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:48:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:48:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:48:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:48:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:49:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:49:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:49:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:49:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:49:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:49:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:49:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:49:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:49:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:49:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:50:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:50:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:50:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:50:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:50:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:50:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:50:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:50:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:50:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:51:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:51:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:51:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:51:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:51:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:51:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:51:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:51:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:51:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:52:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:52:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:52:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:52:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:52:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:52:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:52:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:52:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:52:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:52:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:53:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:53:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:53:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:53:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:53:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:53:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:53:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:53:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:53:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:54:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:54:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:54:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:54:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:54:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:54:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:54:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:54:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:54:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:55:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:55:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:55:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:55:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:55:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:55:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:55:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:55:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:55:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:55:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:56:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:56:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:56:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:56:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:56:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:56:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:56:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:56:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:56:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:57:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:57:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:57:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:57:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:57:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:57:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:57:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:57:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:57:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:58:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:58:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:58:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:58:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:58:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:58:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:58:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:58:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:58:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:58:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:59:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:59:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:59:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:59:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:59:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:59:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:59:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 02:59:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 02:59:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 02:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 02:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 02:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:00:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:00:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:00:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:00:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:00:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:00:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:00:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:00:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:00:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:01:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:01:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:01:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:01:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:01:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:01:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:01:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:01:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:01:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:02:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:02:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:02:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:02:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:02:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:02:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:02:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:02:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:02:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:02:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:03:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:03:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:03:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:03:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:03:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:03:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:03:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:03:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:03:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:04:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:04:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:04:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:04:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:04:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:04:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:04:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:04:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:04:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:05:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:05:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:05:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:05:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:05:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:05:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:05:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:05:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:05:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:05:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:06:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:06:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:06:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:06:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:06:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:06:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:06:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:06:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:06:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:07:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:07:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:07:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:07:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:07:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:07:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:07:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:07:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:07:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:08:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:08:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:08:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:08:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:08:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:08:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:08:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:08:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:08:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:08:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:09:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:09:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:09:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:09:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:09:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:09:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:09:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:09:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:09:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:10:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:10:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:10:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:10:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:10:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:10:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:10:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:10:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:10:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:11:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:11:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:11:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:11:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:11:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:11:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:11:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:11:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:11:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:11:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:12:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:12:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:12:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:12:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:12:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:12:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:12:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:12:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:12:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:13:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:13:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:13:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:13:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:13:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:13:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:13:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:13:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:13:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:14:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:14:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:14:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:14:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:14:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:14:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:14:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:14:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:14:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:14:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:15:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:15:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:15:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:15:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:15:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:15:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:15:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:15:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:15:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:16:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:16:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:16:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:16:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:16:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:16:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:16:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:16:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:16:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:17:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:17:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:17:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:17:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:17:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:17:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:17:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:17:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:17:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:17:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:18:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:18:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:18:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:18:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:18:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:18:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:18:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:18:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:18:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:19:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:19:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:19:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:19:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:19:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:19:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:19:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:19:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:19:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:20:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:20:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:20:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:20:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:20:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:20:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:20:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:20:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:20:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:21:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:21:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:21:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:21:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:21:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:21:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:21:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:21:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:21:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:21:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:22:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:22:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:22:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:22:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:22:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:22:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:22:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:22:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:22:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:23:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:23:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:23:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:23:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:23:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:23:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:23:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:23:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:23:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:24:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:24:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:24:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:24:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:24:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:24:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:24:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:24:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:24:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:24:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:25:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:25:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:25:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:25:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:25:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:25:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:25:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:25:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:25:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:26:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:26:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:26:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:26:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:26:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:26:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:26:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:26:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:26:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:27:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:27:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:27:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:27:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:27:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:27:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:27:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:27:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:27:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:27:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:28:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:28:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:28:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:28:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:28:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:28:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:28:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:28:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:28:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:29:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:29:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:29:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:29:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:29:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:29:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:29:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:29:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:29:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:30:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:30:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:30:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:30:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:30:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:30:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:30:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:30:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:30:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:30:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:31:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:31:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:31:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:31:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:31:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:31:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:31:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:31:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:31:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:32:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:32:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:32:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:32:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:32:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:32:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:32:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:32:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:32:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:33:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:33:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:33:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:33:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:33:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:33:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:33:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:33:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:33:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:33:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:34:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:34:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:34:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:34:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:34:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:34:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:34:45 rdma06 chronyd[1034]: Selected source 10.18.100.10
Apr  8 03:34:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:34:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:34:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:35:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:35:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:35:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:35:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:35:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:35:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:35:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:35:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:35:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:36:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:36:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:36:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:36:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:36:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:36:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:36:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:36:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:36:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:37:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:37:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:37:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:37:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:37:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:37:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:37:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:37:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:37:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:37:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:38:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:38:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:38:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:38:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:38:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:38:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:38:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:38:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:38:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:39:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:39:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:39:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:39:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:39:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:39:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:39:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:39:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:39:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:40:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:40:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:40:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:40:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:40:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:40:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:40:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:40:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:40:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:40:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:41:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:41:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:41:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:41:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:41:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:41:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:41:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:41:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:41:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:42:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:42:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:42:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:42:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:42:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:42:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:42:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:42:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:42:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:43:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:43:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:43:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:43:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:43:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:43:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:43:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:43:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:43:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:43:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:44:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:44:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:44:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:44:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:44:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:44:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:44:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:44:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:44:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:45:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:45:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:45:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:45:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:45:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:45:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:45:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:45:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:45:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:46:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:46:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:46:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:46:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:46:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:46:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:46:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:46:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:46:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:46:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:47:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:47:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:47:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:47:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:47:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:47:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:47:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:47:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:47:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:48:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:48:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:48:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:48:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:48:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:48:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:48:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:48:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:48:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:49:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:49:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:49:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:49:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:49:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:49:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:49:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:49:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:49:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:49:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:50:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:50:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:50:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:50:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:50:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:50:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:50:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:50:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:50:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:51:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:51:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:51:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:51:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:51:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:51:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:51:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:51:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:51:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:52:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:52:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:52:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:52:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:52:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:52:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:52:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:52:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:52:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:53:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:53:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:53:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:53:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:53:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:53:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:53:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:53:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:53:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:53:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:54:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:54:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:54:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:54:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:54:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:54:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:54:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:54:46 rdma06 systemd[1]: Starting dnf makecache...
Apr  8 03:54:46 rdma06 dnf[616050]: Updating Subscription Management repositories.
Apr  8 03:54:46 rdma06 dnf[616050]: Unable to read consumer identity
Apr  8 03:54:46 rdma06 dnf[616050]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  8 03:54:46 rdma06 dnf[616050]: Metadata cache refreshed recently.
Apr  8 03:54:46 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  8 03:54:46 rdma06 systemd[1]: Finished dnf makecache.
Apr  8 03:54:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:54:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:55:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:55:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:55:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:55:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:55:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:55:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:55:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:55:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:55:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:56:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:56:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:56:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:56:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:56:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:56:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:56:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:56:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:56:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:56:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:57:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:57:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:57:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:57:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:57:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:57:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:57:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:57:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:57:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:58:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:58:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:58:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:58:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:58:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:58:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:58:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:58:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:58:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:59:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:59:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:59:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:59:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:59:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:59:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:59:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 03:59:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:59:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 03:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 03:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 03:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 03:59:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:00:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:00:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:00:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:00:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:00:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:00:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:00:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:00:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:00:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:01:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:01:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:01:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:01:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:01:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:01:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:01:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:01:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:01:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:02:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:02:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:02:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:02:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:02:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:02:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:02:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:02:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:02:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:02:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:03:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:03:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:03:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:03:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:03:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:03:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:03:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:03:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:03:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:04:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:04:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:04:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:04:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:04:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:04:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:04:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:04:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:04:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:05:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:05:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:05:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:05:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:05:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:05:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:05:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:05:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:05:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:05:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:06:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:06:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:06:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:06:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:06:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:06:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:06:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:06:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:06:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:07:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:07:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:07:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:07:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:07:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:07:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:07:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:07:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:07:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:08:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:08:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:08:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:08:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:08:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:08:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:08:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:08:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:08:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:09:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:09:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:09:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:09:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:09:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:09:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:09:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:09:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:09:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:09:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:10:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:10:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:10:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:10:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:10:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:10:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:10:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:10:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:10:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:11:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:11:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:11:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:11:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:11:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:11:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:11:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:11:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:11:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:12:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:12:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:12:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:12:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:12:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:12:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:12:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:12:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:12:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:12:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:13:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:13:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:13:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:13:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:13:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:13:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:13:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:13:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:13:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:14:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:14:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:14:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:14:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:14:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:14:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:14:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:14:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:14:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:15:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:15:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:15:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:15:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:15:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:15:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:15:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:15:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:15:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:15:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:16:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:16:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:16:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:16:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:16:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:16:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:16:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:16:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:16:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:17:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:17:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:17:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:17:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:17:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:17:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:17:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:17:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:17:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:18:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:18:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:18:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:18:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:18:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:18:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:18:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:18:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:18:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:18:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:19:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:19:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:19:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:19:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:19:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:19:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:19:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:19:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:19:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:20:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:20:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:20:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:20:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:20:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:20:32 rdma06 NetworkManager[1025]: <info>  [1680942032.7921] dhcp4 (eno1): state changed new lease, address=10.73.131.45
Apr  8 04:20:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:20:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:20:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:20:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:21:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:21:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:21:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:21:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:21:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:21:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:21:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:21:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:21:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:21:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:22:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:22:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:22:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:22:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:22:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:22:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:22:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:22:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:22:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:23:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:23:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:23:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:23:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:23:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:23:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:23:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:23:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:23:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:24:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:24:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:24:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:24:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:24:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:24:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:24:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:24:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:24:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:24:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:25:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:25:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:25:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:25:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:25:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:25:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:25:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:25:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:25:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:26:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:26:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:26:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:26:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:26:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:26:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:26:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:26:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:26:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:27:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:27:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:27:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:27:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:27:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:27:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:27:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:27:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:27:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:28:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:28:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:28:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:28:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:28:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:28:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:28:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:28:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:28:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:28:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:29:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:29:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:29:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:29:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:29:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:29:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:29:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:29:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:29:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:30:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:30:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:30:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:30:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:30:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:30:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:30:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:30:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:30:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:31:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:31:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:31:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:31:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:31:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:31:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:31:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:31:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:31:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:31:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:32:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:32:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:32:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:32:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:32:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:32:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:32:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:32:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:32:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:33:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:33:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:33:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:33:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:33:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:33:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:33:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:33:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:33:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:34:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:34:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:34:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:34:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:34:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:34:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:34:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:34:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:34:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:34:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:35:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:35:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:35:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:35:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:35:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:35:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:35:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:35:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:35:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:36:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:36:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:36:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:36:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:36:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:36:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:36:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:36:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:36:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:37:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:37:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:37:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:37:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:37:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:37:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:37:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:37:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:37:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:37:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:38:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:38:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:38:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:38:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:38:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:38:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:38:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:38:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:38:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:39:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:39:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:39:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:39:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:39:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:39:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:39:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:39:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:39:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:40:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:40:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:40:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:40:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:40:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:40:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:40:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:40:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:40:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:40:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:41:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:41:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:41:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:41:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:41:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:41:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:41:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:41:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:41:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:42:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:42:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:42:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:42:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:42:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:42:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:42:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:42:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:42:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:43:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:43:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:43:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:43:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:43:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:43:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:43:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:43:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:43:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:44:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:44:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:44:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:44:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:44:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:44:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:44:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:44:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:44:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:44:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:45:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:45:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:45:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:45:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:45:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:45:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:45:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:45:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:45:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:46:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:46:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:46:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:46:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:46:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:46:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:46:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:46:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:46:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:47:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:47:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:47:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:47:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:47:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:47:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:47:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:47:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:47:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:47:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:48:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:48:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:48:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:48:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:48:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:48:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:48:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:48:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:48:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:49:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:49:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:49:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:49:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:49:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:49:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:49:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:49:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:49:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:50:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:50:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:50:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:50:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:50:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:50:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:50:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:50:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:50:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:50:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:51:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:51:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:51:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:51:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:51:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:51:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:51:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:51:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:51:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:52:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:52:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:52:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:52:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:52:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:52:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:52:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:52:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:52:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:53:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:53:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:53:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:53:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:53:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:53:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:53:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:53:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:53:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:53:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:54:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:54:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:54:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:54:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:54:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:54:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:54:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:54:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:54:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:55:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:55:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:55:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:55:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:55:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:55:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:55:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:55:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:55:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:56:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:56:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:56:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:56:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:56:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:56:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:56:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:56:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:56:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:56:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:57:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:57:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:57:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:57:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:57:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:57:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:57:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:57:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:57:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:58:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:58:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:58:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:58:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:58:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:58:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:58:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:58:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:58:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:59:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:59:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:59:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:59:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:59:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:59:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:59:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 04:59:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 04:59:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 04:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 04:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 04:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:00:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:00:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:00:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:00:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:00:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:00:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:00:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:00:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:00:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:00:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:01:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:01:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:01:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:01:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:01:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:01:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:01:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:01:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:01:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:02:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  8 05:02:01 rdma06 dnf[618779]: Updating Subscription Management repositories.
Apr  8 05:02:01 rdma06 dnf[618779]: Unable to read consumer identity
Apr  8 05:02:01 rdma06 dnf[618779]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  8 05:02:03 rdma06 dnf[618779]: Copr repo for fmf owned by psss                 2.2 kB/s | 3.3 kB     00:01
Apr  8 05:02:03 rdma06 dnf[618779]: beaker-AppStream                                 63 kB/s | 3.2 kB     00:00
Apr  8 05:02:03 rdma06 dnf[618779]: beaker-AppStream-debuginfo                       49 kB/s | 1.5 kB     00:00
Apr  8 05:02:03 rdma06 dnf[618779]: beaker-BaseOS                                   147 kB/s | 2.7 kB     00:00
Apr  8 05:02:03 rdma06 dnf[618779]: beaker-BaseOS-debuginfo                          45 kB/s | 1.5 kB     00:00
Apr  8 05:02:03 rdma06 dnf[618779]: beaker-CRB                                       80 kB/s | 2.7 kB     00:00
Apr  8 05:02:03 rdma06 dnf[618779]: beaker-CRB-debuginfo                             70 kB/s | 1.5 kB     00:00
Apr  8 05:02:03 rdma06 dnf[618779]: beaker-HighAvailability                          67 kB/s | 2.8 kB     00:00
Apr  8 05:02:03 rdma06 dnf[618779]: beaker-HighAvailability-debuginfo                50 kB/s | 1.5 kB     00:00
Apr  8 05:02:03 rdma06 dnf[618779]: beaker-NFV                                       71 kB/s | 2.7 kB     00:00
Apr  8 05:02:03 rdma06 dnf[618779]: beaker-NFV-debuginfo                             48 kB/s | 1.5 kB     00:00
Apr  8 05:02:03 rdma06 dnf[618779]: beaker-RT                                        32 kB/s | 2.7 kB     00:00
Apr  8 05:02:03 rdma06 dnf[618779]: beaker-RT-debuginfo                              57 kB/s | 1.5 kB     00:00
Apr  8 05:02:03 rdma06 dnf[618779]: beaker-ResilientStorage                          56 kB/s | 2.8 kB     00:00
Apr  8 05:02:03 rdma06 dnf[618779]: beaker-ResilientStorage-debuginfo                32 kB/s | 1.5 kB     00:00
Apr  8 05:02:03 rdma06 dnf[618779]: beaker-SAP                                       77 kB/s | 2.7 kB     00:00
Apr  8 05:02:04 rdma06 dnf[618779]: beaker-SAP-debuginfo                             12 kB/s | 1.5 kB     00:00
Apr  8 05:02:04 rdma06 dnf[618779]: beaker-SAPHANA                                   47 kB/s | 2.7 kB     00:00
Apr  8 05:02:04 rdma06 dnf[618779]: beaker-SAPHANA-debuginfo                         28 kB/s | 1.5 kB     00:00
Apr  8 05:02:04 rdma06 systemd-journald[790]: Data hash table of /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal has a fill level at 75.0 (163750 of 218332 items, 75497472 file size, 461 bytes per hash table item), suggesting rotation.
Apr  8 05:02:04 rdma06 systemd-journald[790]: /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal: Journal header limits reached or header out-of-date, rotating.
Apr  8 05:02:04 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr  8 05:02:04 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr  8 05:02:04 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr  8 05:02:04 rdma06 dnf[618779]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr  8 05:02:05 rdma06 dnf[618779]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr  8 05:02:05 rdma06 dnf[618779]: Metadata cache created.
Apr  8 05:02:05 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  8 05:02:05 rdma06 systemd[1]: Finished dnf makecache.
Apr  8 05:02:05 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.383s CPU time.
Apr  8 05:02:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:02:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:02:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:02:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:02:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:02:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:02:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:02:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:02:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:03:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:03:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:03:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:03:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:03:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:03:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:03:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:03:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:03:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:03:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:04:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:04:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:04:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:04:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:04:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:04:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:04:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:04:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:04:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:05:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:05:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:05:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:05:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:05:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:05:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:05:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:05:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:05:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:06:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:06:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:06:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:06:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:06:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:06:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:06:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:06:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:06:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:06:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:07:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:07:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:07:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:07:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:07:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:07:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:07:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:07:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:07:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:08:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:08:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:08:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:08:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:08:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:08:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:08:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:08:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:08:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:09:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:09:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:09:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:09:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:09:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:09:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:09:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:09:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:09:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:09:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:10:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:10:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:10:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:10:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:10:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:10:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:10:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:10:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:10:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:11:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:11:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:11:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:11:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:11:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:11:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:11:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:11:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:11:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:12:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:12:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:12:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:12:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:12:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:12:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:12:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:12:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:12:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:12:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:13:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:13:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:13:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:13:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:13:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:13:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:13:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:13:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:13:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:14:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:14:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:14:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:14:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:14:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:14:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:14:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:14:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:14:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:15:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:15:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:15:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:15:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:15:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:15:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:15:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:15:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:15:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:16:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:16:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:16:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:16:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:16:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:16:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:16:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:16:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:16:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:16:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:17:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:17:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:17:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:17:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:17:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:17:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:17:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:17:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:17:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:18:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:18:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:18:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:18:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:18:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:18:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:18:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:18:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:18:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:19:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:19:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:19:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:19:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:19:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:19:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:19:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:19:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:19:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:19:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:20:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:20:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:20:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:20:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:20:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:20:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:20:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:20:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:20:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:21:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:21:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:21:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:21:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:21:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:21:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:21:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:21:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:21:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:22:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:22:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:22:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:22:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:22:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:22:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:22:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:22:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:22:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:22:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:23:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:23:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:23:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:23:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:23:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:23:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:23:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:23:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:23:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:24:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:24:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:24:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:24:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:24:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:24:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:24:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:24:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:24:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:25:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:25:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:25:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:25:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:25:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:25:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:25:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:25:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:25:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:25:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:26:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:26:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:26:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:26:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:26:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:26:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:26:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:26:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:26:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:27:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:27:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:27:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:27:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:27:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:27:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:27:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:27:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:27:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:28:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:28:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:28:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:28:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:28:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:28:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:28:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:28:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:28:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:28:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:29:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:29:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:29:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:29:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:29:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:29:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:29:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:29:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:29:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:30:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:30:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:30:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:30:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:30:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:30:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:30:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:30:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:30:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:31:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:31:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:31:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:31:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:31:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:31:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:31:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:31:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:31:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:31:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:32:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:32:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:32:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:32:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:32:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:32:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:32:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:32:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:32:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:33:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:33:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:33:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:33:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:33:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:33:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:33:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:33:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:33:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:34:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:34:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:34:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:34:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:34:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:34:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:34:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:34:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:34:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:35:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:35:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:35:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:35:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:35:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:35:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:35:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:35:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:35:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:35:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:36:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:36:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:36:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:36:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:36:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:36:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:36:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:36:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:36:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:37:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:37:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:37:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:37:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:37:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:37:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:37:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:37:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:37:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:38:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:38:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:38:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:38:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:38:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:38:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:38:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:38:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:38:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:38:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:39:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:39:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:39:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:39:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:39:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:39:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:39:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:39:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:39:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:40:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:40:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:40:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:40:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:40:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:40:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:40:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:40:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:40:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:41:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:41:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:41:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:41:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:41:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:41:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:41:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:41:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:41:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:41:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:42:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:42:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:42:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:42:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:42:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:42:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:42:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:42:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:42:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:43:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:43:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:43:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:43:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:43:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:43:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:43:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:43:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:43:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:44:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:44:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:44:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:44:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:44:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:44:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:44:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:44:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:44:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:44:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:45:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:45:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:45:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:45:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:45:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:45:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:45:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:45:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:45:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:46:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:46:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:46:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:46:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:46:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:46:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:46:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:46:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:46:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:47:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:47:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:47:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:47:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:47:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:47:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:47:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:47:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:47:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:47:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:48:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:48:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:48:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:48:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:48:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:48:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:48:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:48:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:48:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:49:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:49:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:49:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:49:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:49:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:49:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:49:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:49:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:49:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:50:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:50:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:50:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:50:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:50:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:50:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:50:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:50:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:50:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:51:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:51:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:51:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:51:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:51:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:51:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:51:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:51:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:51:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:51:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:52:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:52:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:52:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:52:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:52:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:52:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:52:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:52:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:52:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:53:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:53:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:53:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:53:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:53:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:53:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:53:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:53:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:53:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:54:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:54:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:54:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:54:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:54:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:54:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:54:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:54:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:54:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:54:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:55:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:55:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:55:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:55:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:55:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:55:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:55:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:55:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:55:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:56:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:56:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:56:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:56:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:56:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:56:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:56:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:56:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:56:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:57:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:57:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:57:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:57:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:57:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:57:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:57:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:57:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:57:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:57:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:58:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:58:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:58:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:58:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:58:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:58:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:58:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:58:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:58:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:59:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:59:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:59:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:59:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:59:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:59:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:59:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 05:59:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 05:59:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 05:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 05:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 05:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:00:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:00:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:00:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:00:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:00:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:00:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:00:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:00:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:00:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:00:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:01:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:01:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:01:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:01:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:01:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:01:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:01:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:01:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:01:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:02:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:02:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:02:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:02:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:02:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:02:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:02:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:02:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:02:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:03:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:03:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:03:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:03:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:03:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:03:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:03:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:03:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:03:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:03:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:04:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:04:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:04:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:04:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:04:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:04:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:04:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:04:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:04:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:05:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:05:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:05:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:05:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:05:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:05:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:05:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:05:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:05:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:06:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:06:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:06:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:06:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:06:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:06:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:06:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:06:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:06:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:07:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:07:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:07:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:07:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:07:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:07:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:07:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:07:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:07:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:07:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:08:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:08:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:08:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:08:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:08:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:08:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:08:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:08:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:08:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:09:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:09:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:09:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:09:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:09:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:09:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:09:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:09:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:09:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:10:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:10:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:10:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:10:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:10:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:10:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:10:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:10:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:10:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:10:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:11:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:11:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:11:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:11:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:11:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:11:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:11:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:11:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:11:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:12:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:12:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:12:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:12:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:12:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:12:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:12:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:12:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:12:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:13:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:13:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:13:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:13:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:13:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:13:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:13:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:13:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:13:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:13:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:14:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:14:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:14:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:14:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:14:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:14:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:14:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:14:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:14:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:15:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:15:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:15:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:15:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:15:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:15:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:15:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:15:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:15:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:16:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:16:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:16:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:16:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:16:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:16:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:16:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:16:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:16:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:16:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:17:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  8 06:17:01 rdma06 dnf[621838]: Updating Subscription Management repositories.
Apr  8 06:17:01 rdma06 dnf[621838]: Unable to read consumer identity
Apr  8 06:17:01 rdma06 dnf[621838]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  8 06:17:01 rdma06 dnf[621838]: Metadata cache refreshed recently.
Apr  8 06:17:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  8 06:17:01 rdma06 systemd[1]: Finished dnf makecache.
Apr  8 06:17:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:17:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:17:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:17:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:17:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:17:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:17:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:17:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:17:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:18:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:18:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:18:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:18:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:18:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:18:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:18:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:18:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:18:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:19:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:19:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:19:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:19:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:19:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:19:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:19:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:19:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:19:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:19:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:20:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:20:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:20:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:20:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:20:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:20:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:20:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:20:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:20:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:21:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:21:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:21:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:21:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:21:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:21:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:21:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:21:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:21:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:22:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:22:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:22:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:22:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:22:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:22:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:22:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:22:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:22:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:23:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:23:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:23:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:23:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:23:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:23:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:23:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:23:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:23:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:23:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:24:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:24:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:24:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:24:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:24:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:24:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:24:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:24:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:24:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:25:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:25:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:25:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:25:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:25:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:25:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:25:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:25:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:25:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:26:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:26:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:26:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:26:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:26:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:26:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:26:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:26:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:26:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:26:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:27:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:27:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:27:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:27:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:27:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:27:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:27:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:27:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:27:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:28:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:28:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:28:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:28:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:28:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:28:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:28:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:28:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:28:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:29:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:29:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:29:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:29:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:29:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:29:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:29:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:29:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:29:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:29:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:30:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:30:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:30:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:30:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:30:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:30:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:30:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:30:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:30:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:31:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:31:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:31:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:31:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:31:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:31:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:31:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:31:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:31:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:32:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:32:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:32:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:32:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:32:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:32:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:32:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:32:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:32:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:32:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:33:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:33:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:33:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:33:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:33:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:33:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:33:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:33:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:33:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:34:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:34:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:34:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:34:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:34:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:34:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:34:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:34:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:34:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:35:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:35:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:35:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:35:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:35:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:35:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:35:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:35:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:35:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:35:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:36:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:36:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:36:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:36:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:36:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:36:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:36:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:36:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:36:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:37:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:37:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:37:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:37:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:37:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:37:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:37:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:37:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:37:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:38:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:38:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:38:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:38:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:38:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:38:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:38:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:38:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:38:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:38:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:39:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:39:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:39:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:39:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:39:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:39:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:39:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:39:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:39:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:40:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:40:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:40:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:40:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:40:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:40:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:40:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:40:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:40:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:41:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:41:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:41:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:41:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:41:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:41:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:41:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:41:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:41:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:42:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:42:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:42:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:42:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:42:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:42:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:42:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:42:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:42:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:42:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:43:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:43:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:43:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:43:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:43:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:43:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:43:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:43:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:43:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:44:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:44:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:44:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:44:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:44:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:44:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:44:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:44:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:44:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:45:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:45:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:45:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:45:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:45:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:45:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:45:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:45:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:45:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:45:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:46:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:46:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:46:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:46:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:46:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:46:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:46:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:46:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:46:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:47:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:47:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:47:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:47:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:47:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:47:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:47:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:47:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:47:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:48:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:48:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:48:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:48:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:48:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:48:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:48:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:48:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:48:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:48:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:49:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:49:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:49:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:49:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:49:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:49:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:49:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:49:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:49:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:50:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:50:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:50:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:50:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:50:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:50:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:50:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:50:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:50:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:51:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:51:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:51:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:51:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:51:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:51:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:51:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:51:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:51:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:51:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:52:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:52:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:52:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:52:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:52:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:52:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:52:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:52:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:52:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:53:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:53:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:53:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:53:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:53:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:53:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:53:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:53:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:53:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:54:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:54:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:54:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:54:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:54:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:54:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:54:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:54:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:54:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:54:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:55:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:55:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:55:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:55:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:55:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:55:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:55:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:55:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:55:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:56:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:56:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:56:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:56:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:56:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:56:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:56:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:56:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:56:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:57:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:57:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:57:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:57:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:57:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:57:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:57:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:57:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:57:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:58:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:58:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:58:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:58:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:58:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:58:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:58:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:58:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:58:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:58:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:59:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:59:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:59:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:59:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:59:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:59:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:59:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 06:59:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 06:59:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 06:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 06:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 06:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:00:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:00:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:00:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:00:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:00:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:00:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:00:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:00:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:00:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:01:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:01:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:01:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:01:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:01:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:01:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:01:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:01:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:01:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:01:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:02:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:02:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:02:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:02:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:02:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:02:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:02:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:02:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:02:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:03:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:03:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:03:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:03:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:03:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:03:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:03:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:03:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:03:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:04:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:04:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:04:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:04:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:04:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:04:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:04:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:04:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:04:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:04:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:05:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:05:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:05:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:05:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:05:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:05:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:05:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:05:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:05:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:06:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:06:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:06:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:06:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:06:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:06:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:06:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:06:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:06:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:07:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:07:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:07:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:07:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:07:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:07:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:07:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:07:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:07:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:07:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:08:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:08:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:08:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:08:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:08:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:08:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:08:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:08:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:08:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:09:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:09:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:09:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:09:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:09:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:09:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:09:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:09:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:09:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:10:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:10:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:10:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:10:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:10:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:10:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:10:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:10:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:10:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:10:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:11:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:11:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:11:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:11:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:11:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:11:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:11:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:11:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:11:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:12:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:12:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:12:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:12:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:12:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:12:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:12:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:12:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:12:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:13:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:13:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:13:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:13:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:13:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:13:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:13:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:13:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:13:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:14:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:14:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:14:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:14:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:14:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:14:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:14:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:14:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:14:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:14:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:15:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:15:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:15:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:15:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:15:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:15:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:15:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:15:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:15:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:16:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:16:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:16:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:16:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:16:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:16:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:16:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:16:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:16:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:17:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:17:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:17:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:17:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:17:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:17:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:17:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:17:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:17:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:17:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:18:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:18:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:18:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:18:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:18:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:18:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:18:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:18:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:18:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:19:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:19:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:19:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:19:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:19:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:19:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:19:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:19:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:19:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:20:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:20:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:20:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:20:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:20:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:20:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:20:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:20:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:20:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:20:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:21:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:21:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:21:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:21:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:21:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:21:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:21:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:21:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:21:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:22:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:22:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:22:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:22:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:22:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:22:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:22:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:22:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:22:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:23:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:23:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:23:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:23:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:23:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:23:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:23:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:23:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:23:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:23:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:24:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:24:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:24:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:24:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:24:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:24:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:24:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:24:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:24:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:25:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:25:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:25:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:25:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:25:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:25:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:25:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:25:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:25:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:26:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:26:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:26:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:26:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:26:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:26:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:26:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:26:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:26:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:26:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:27:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:27:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:27:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:27:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:27:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:27:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:27:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:27:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:27:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:28:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:28:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:28:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:28:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:28:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:28:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:28:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:28:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:28:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:29:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:29:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:29:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:29:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:29:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:29:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:29:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:29:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:29:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:30:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:30:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:30:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:30:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:30:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:30:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:30:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:30:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:30:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:30:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:31:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:31:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:31:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:31:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:31:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:31:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:31:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:31:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:31:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:32:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:32:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:32:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:32:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:32:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:32:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:32:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:32:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:32:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:33:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:33:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:33:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:33:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:33:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:33:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:33:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:33:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:33:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:33:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:34:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:34:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:34:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:34:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:34:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:34:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:34:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:34:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:34:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:35:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:35:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:35:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:35:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:35:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:35:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:35:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:35:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:35:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:36:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:36:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:36:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:36:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:36:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:36:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:36:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:36:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:36:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:36:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:37:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:37:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:37:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:37:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:37:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:37:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:37:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:37:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:37:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:38:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:38:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:38:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:38:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:38:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:38:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:38:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:38:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:38:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:39:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:39:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:39:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:39:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:39:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:39:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:39:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:39:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:39:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:39:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:40:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:40:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:40:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:40:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:40:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:40:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:40:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:40:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:40:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:41:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:41:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:41:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:41:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:41:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:41:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:41:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:41:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:41:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:42:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:42:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:42:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:42:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:42:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:42:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:42:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:42:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:42:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:42:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:43:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:43:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:43:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:43:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:43:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:43:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:43:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:43:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:43:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:44:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:44:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:44:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:44:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:44:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:44:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:44:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:44:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:44:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:45:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:45:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:45:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:45:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:45:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:45:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:45:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:45:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:45:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:45:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:46:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:46:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:46:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:46:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:46:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:46:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:46:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:46:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:46:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:47:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:47:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:47:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:47:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:47:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:47:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:47:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:47:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:47:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:48:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:48:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:48:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:48:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:48:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:48:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:48:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:48:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:48:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:49:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:49:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:49:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:49:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:49:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:49:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:49:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:49:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:49:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:49:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:50:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:50:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:50:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:50:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:50:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:50:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:50:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:50:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:50:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:51:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:51:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:51:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:51:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:51:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:51:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:51:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:51:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:51:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:52:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:52:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:52:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:52:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:52:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:52:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:52:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:52:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:52:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:52:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:53:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:53:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:53:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:53:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:53:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:53:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:53:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:53:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:53:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:54:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:54:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:54:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:54:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:54:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:54:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:54:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:54:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:54:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:55:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:55:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:55:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:55:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:55:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:55:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:55:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:55:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:55:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:55:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:56:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:56:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:56:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:56:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:56:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:56:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:56:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:56:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:56:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:57:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:57:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:57:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:57:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:57:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:57:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:57:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:57:46 rdma06 systemd[1]: Starting dnf makecache...
Apr  8 07:57:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:57:46 rdma06 dnf[625908]: Updating Subscription Management repositories.
Apr  8 07:57:46 rdma06 dnf[625908]: Unable to read consumer identity
Apr  8 07:57:46 rdma06 dnf[625908]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  8 07:57:46 rdma06 dnf[625908]: Metadata cache refreshed recently.
Apr  8 07:57:46 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  8 07:57:46 rdma06 systemd[1]: Finished dnf makecache.
Apr  8 07:57:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:58:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:58:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:58:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:58:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:58:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:58:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:58:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:58:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:58:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:58:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:59:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:59:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:59:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:59:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:59:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:59:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:59:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 07:59:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 07:59:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 07:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 07:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 07:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:00:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:00:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:00:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:00:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:00:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:00:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:00:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:00:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:00:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:01:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:01:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:01:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:01:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:01:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:01:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:01:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:01:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:01:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:01:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:02:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:02:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:02:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:02:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:02:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:02:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:02:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:02:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:02:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:03:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:03:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:03:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:03:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:03:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:03:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:03:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:03:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:03:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:04:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:04:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:04:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:04:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:04:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:04:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:04:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:04:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:04:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:05:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:05:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:05:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:05:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:05:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:05:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:05:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:05:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:05:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:05:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:06:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:06:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:06:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:06:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:06:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:06:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:06:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:06:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:06:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:07:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:07:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:07:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:07:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:07:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:07:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:07:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:07:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:07:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:08:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:08:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:08:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:08:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:08:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:08:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:08:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:08:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:08:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:08:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:09:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:09:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:09:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:09:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:09:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:09:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:09:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:09:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:09:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:10:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:10:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:10:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:10:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:10:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:10:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:10:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:10:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:10:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:11:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:11:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:11:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:11:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:11:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:11:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:11:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:11:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:11:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:11:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:12:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:12:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:12:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:12:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:12:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:12:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:12:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:12:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:12:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:13:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:13:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:13:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:13:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:13:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:13:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:13:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:13:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:13:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:14:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:14:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:14:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:14:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:14:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:14:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:14:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:14:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:14:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:14:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:15:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:15:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:15:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:15:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:15:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:15:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:15:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:15:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:15:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:16:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:16:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:16:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:16:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:16:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:16:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:16:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:16:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:16:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:17:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:17:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:17:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:17:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:17:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:17:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:17:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:17:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:17:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:17:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:18:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:18:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:18:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:18:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:18:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:18:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:18:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:18:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:18:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:19:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:19:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:19:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:19:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:19:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:19:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:19:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:19:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:19:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:20:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:20:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:20:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:20:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:20:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:20:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:20:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:20:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:20:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:21:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:21:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:21:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:21:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:21:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:21:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:21:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:21:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:21:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:21:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:22:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:22:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:22:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:22:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:22:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:22:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:22:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:22:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:22:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:23:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:23:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:23:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:23:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:23:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:23:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:23:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:23:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:23:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:24:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:24:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:24:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:24:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:24:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:24:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:24:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:24:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:24:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:24:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:25:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:25:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:25:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:25:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:25:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:25:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:25:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:25:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:25:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:26:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:26:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:26:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:26:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:26:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:26:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:26:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:26:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:26:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:27:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:27:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:27:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:27:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:27:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:27:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:27:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:27:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:27:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:27:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:28:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:28:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:28:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:28:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:28:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:28:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:28:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:28:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:28:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:29:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:29:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:29:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:29:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:29:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:29:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:29:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:29:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:29:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:30:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:30:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:30:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:30:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:30:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:30:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:30:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:30:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:30:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:30:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:31:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:31:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:31:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:31:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:31:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:31:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:31:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:31:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:31:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:32:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:32:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:32:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:32:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:32:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:32:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:32:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:32:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:32:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:33:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:33:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:33:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:33:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:33:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:33:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:33:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:33:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:33:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:33:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:34:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:34:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:34:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:34:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:34:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:34:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:34:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:34:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:34:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:35:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:35:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:35:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:35:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:35:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:35:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:35:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:35:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:35:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:36:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:36:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:36:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:36:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:36:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:36:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:36:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:36:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:36:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:37:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:37:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:37:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:37:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:37:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:37:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:37:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:37:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:37:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:37:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:38:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:38:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:38:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:38:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:38:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:38:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:38:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:38:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:38:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:39:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:39:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:39:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:39:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:39:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:39:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:39:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:39:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:39:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:40:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:40:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:40:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:40:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:40:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:40:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:40:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:40:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:40:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:40:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:41:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:41:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:41:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:41:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:41:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:41:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:41:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:41:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:41:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:42:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:42:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:42:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:42:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:42:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:42:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:42:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:42:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:42:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:43:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:43:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:43:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:43:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:43:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:43:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:43:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:43:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:43:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:43:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:44:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:44:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:44:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:44:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:44:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:44:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:44:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:44:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:44:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:45:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:45:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:45:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:45:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:45:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:45:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:45:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:45:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:45:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:46:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:46:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:46:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:46:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:46:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:46:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:46:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:46:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:46:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:46:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:47:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:47:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:47:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:47:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:47:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:47:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:47:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:47:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:47:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:48:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:48:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:48:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:48:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:48:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:48:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:48:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:48:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:48:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:49:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:49:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:49:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:49:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:49:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:49:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:49:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:49:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:49:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:49:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:50:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:50:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:50:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:50:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:50:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:50:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:50:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:50:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:50:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:51:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:51:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:51:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:51:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:51:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:51:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:51:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:51:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:51:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:52:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:52:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:52:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:52:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:52:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:52:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:52:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:52:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:52:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:52:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:53:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:53:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:53:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:53:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:53:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:53:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:53:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:53:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:53:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:54:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:54:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:54:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:54:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:54:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:54:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:54:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:54:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:54:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:55:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:55:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:55:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:55:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:55:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:55:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:55:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:55:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:55:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:56:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:56:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:56:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:56:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:56:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:56:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:56:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:56:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:56:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:56:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:57:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:57:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:57:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:57:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:57:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:57:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:57:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:57:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:57:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:58:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:58:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:58:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:58:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:58:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:58:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:58:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:58:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:58:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:59:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:59:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:59:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:59:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:59:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:59:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:59:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:59:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 08:59:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 08:59:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 08:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 08:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 08:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:00:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:00:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:00:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:00:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:00:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:00:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:00:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:00:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:00:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:01:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:01:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:01:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:01:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:01:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:01:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:01:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:01:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:01:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:02:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:02:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:02:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:02:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:02:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:02:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:02:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:02:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:02:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:02:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:03:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:03:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:03:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:03:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:03:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:03:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:03:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:03:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:03:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:04:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:04:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:04:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:04:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:04:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:04:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:04:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:04:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:04:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:05:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:05:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:05:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:05:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:05:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:05:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:05:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:05:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:05:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:05:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:06:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:06:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:06:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:06:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:06:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:06:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:06:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:06:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:06:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:07:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:07:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:07:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:07:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:07:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:07:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:07:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:07:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:07:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:08:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:08:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:08:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:08:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:08:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:08:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:08:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:08:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:08:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:08:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:09:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:09:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:09:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:09:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:09:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:09:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:09:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:09:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:09:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:10:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:10:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:10:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:10:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:10:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:10:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:10:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:10:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:10:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:11:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:11:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:11:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:11:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:11:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:11:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:11:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:11:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:11:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:12:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:12:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:12:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:12:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:12:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:12:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:12:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:12:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:12:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:12:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:13:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:13:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:13:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:13:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:13:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:13:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:13:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:13:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:13:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:14:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:14:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:14:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:14:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:14:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:14:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:14:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:14:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:14:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:15:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:15:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:15:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:15:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:15:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:15:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:15:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:15:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:15:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:15:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:16:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:16:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:16:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:16:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:16:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:16:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:16:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:16:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:16:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:17:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:17:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:17:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:17:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:17:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:17:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:17:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:17:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:17:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:18:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:18:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:18:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:18:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:18:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:18:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:18:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:18:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:18:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:18:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:19:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:19:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:19:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:19:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:19:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:19:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:19:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:19:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:19:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:20:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:20:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:20:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:20:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:20:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:20:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:20:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:20:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:20:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:21:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:21:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:21:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:21:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:21:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:21:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:21:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:21:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:21:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:21:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:22:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:22:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:22:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:22:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:22:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:22:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:22:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:22:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:22:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:23:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:23:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:23:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:23:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:23:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:23:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:23:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:23:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:23:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:24:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:24:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:24:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:24:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:24:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:24:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:24:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:24:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:24:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:24:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:25:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:25:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:25:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:25:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:25:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:25:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:25:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:25:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:25:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:26:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:26:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:26:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:26:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:26:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:26:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:26:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:26:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:26:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:27:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:27:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:27:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:27:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:27:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:27:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:27:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:27:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:27:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:28:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:28:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:28:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:28:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:28:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:28:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:28:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:28:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:28:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:28:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:29:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:29:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:29:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:29:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:29:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:29:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:29:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:29:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:29:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:30:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:30:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:30:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:30:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:30:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:30:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:30:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:30:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:30:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:31:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:31:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:31:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:31:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:31:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:31:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:31:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:31:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:31:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:31:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:32:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:32:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:32:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:32:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:32:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:32:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:32:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:32:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:32:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:33:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:33:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:33:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:33:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:33:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:33:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:33:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:33:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:33:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:34:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:34:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:34:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:34:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:34:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:34:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:34:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:34:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:34:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:34:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:35:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:35:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:35:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:35:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:35:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:35:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:35:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:35:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:35:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:36:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:36:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:36:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:36:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:36:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:36:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:36:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:36:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:36:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:37:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:37:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:37:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:37:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:37:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:37:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:37:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:37:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:37:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:37:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:38:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:38:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:38:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:38:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:38:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:38:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:38:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:38:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:38:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:39:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:39:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:39:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:39:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:39:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:39:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:39:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:39:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:39:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:40:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:40:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:40:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:40:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:40:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:40:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:40:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:40:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:40:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:40:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:41:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:41:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:41:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:41:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:41:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:41:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:41:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:41:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:41:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:42:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:42:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:42:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:42:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:42:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:42:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:42:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:42:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:42:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:43:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:43:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:43:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:43:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:43:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:43:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:43:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:43:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:43:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:44:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:44:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:44:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:44:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:44:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:44:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:44:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:44:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:44:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:44:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:45:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:45:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:45:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:45:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:45:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:45:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:45:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:45:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:45:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:46:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:46:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:46:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:46:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:46:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:46:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:46:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:46:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:46:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:47:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:47:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:47:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:47:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:47:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:47:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:47:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:47:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:47:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:47:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:48:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:48:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:48:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:48:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:48:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:48:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:48:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:48:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:48:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:49:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:49:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:49:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:49:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:49:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:49:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:49:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:49:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:49:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:50:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:50:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:50:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:50:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:50:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:50:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:50:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:50:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:50:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:50:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:51:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:51:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:51:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:51:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:51:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:51:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:51:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:51:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:51:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:52:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:52:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:52:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:52:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:52:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:52:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:52:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:52:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:52:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:53:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:53:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:53:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:53:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:53:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:53:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:53:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:53:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:53:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:53:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:54:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:54:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:54:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:54:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:54:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:54:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:54:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:54:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:54:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:55:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:55:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:55:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:55:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:55:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:55:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:55:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:55:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:55:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:56:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  8 09:56:01 rdma06 dnf[630697]: Updating Subscription Management repositories.
Apr  8 09:56:01 rdma06 dnf[630697]: Unable to read consumer identity
Apr  8 09:56:01 rdma06 dnf[630697]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  8 09:56:02 rdma06 dnf[630697]: Copr repo for fmf owned by psss                 2.8 kB/s | 3.3 kB     00:01
Apr  8 09:56:02 rdma06 dnf[630697]: beaker-AppStream                                 41 kB/s | 3.2 kB     00:00
Apr  8 09:56:02 rdma06 dnf[630697]: beaker-AppStream-debuginfo                       36 kB/s | 1.5 kB     00:00
Apr  8 09:56:03 rdma06 dnf[630697]: beaker-BaseOS                                    94 kB/s | 2.7 kB     00:00
Apr  8 09:56:03 rdma06 dnf[630697]: beaker-BaseOS-debuginfo                          41 kB/s | 1.5 kB     00:00
Apr  8 09:56:03 rdma06 dnf[630697]: beaker-CRB                                       59 kB/s | 2.7 kB     00:00
Apr  8 09:56:03 rdma06 dnf[630697]: beaker-CRB-debuginfo                             72 kB/s | 1.5 kB     00:00
Apr  8 09:56:03 rdma06 dnf[630697]: beaker-HighAvailability                          83 kB/s | 2.8 kB     00:00
Apr  8 09:56:03 rdma06 dnf[630697]: beaker-HighAvailability-debuginfo                40 kB/s | 1.5 kB     00:00
Apr  8 09:56:03 rdma06 dnf[630697]: beaker-NFV                                       77 kB/s | 2.7 kB     00:00
Apr  8 09:56:03 rdma06 dnf[630697]: beaker-NFV-debuginfo                             56 kB/s | 1.5 kB     00:00
Apr  8 09:56:03 rdma06 dnf[630697]: beaker-RT                                        69 kB/s | 2.7 kB     00:00
Apr  8 09:56:03 rdma06 dnf[630697]: beaker-RT-debuginfo                              65 kB/s | 1.5 kB     00:00
Apr  8 09:56:03 rdma06 dnf[630697]: beaker-ResilientStorage                          67 kB/s | 2.8 kB     00:00
Apr  8 09:56:03 rdma06 dnf[630697]: beaker-ResilientStorage-debuginfo                60 kB/s | 1.5 kB     00:00
Apr  8 09:56:03 rdma06 dnf[630697]: beaker-SAP                                       96 kB/s | 2.7 kB     00:00
Apr  8 09:56:03 rdma06 dnf[630697]: beaker-SAP-debuginfo                             59 kB/s | 1.5 kB     00:00
Apr  8 09:56:03 rdma06 dnf[630697]: beaker-SAPHANA                                   82 kB/s | 2.7 kB     00:00
Apr  8 09:56:03 rdma06 dnf[630697]: beaker-SAPHANA-debuginfo                         44 kB/s | 1.5 kB     00:00
Apr  8 09:56:04 rdma06 dnf[630697]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr  8 09:56:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:56:05 rdma06 dnf[630697]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr  8 09:56:05 rdma06 dnf[630697]: Metadata cache created.
Apr  8 09:56:05 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  8 09:56:05 rdma06 systemd[1]: Finished dnf makecache.
Apr  8 09:56:05 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.417s CPU time.
Apr  8 09:56:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:56:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:56:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:56:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:56:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:56:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:56:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:56:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:56:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:57:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:57:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:57:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:57:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:57:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:57:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:57:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:57:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:57:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:58:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:58:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:58:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:58:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:58:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:58:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:58:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:58:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:58:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:59:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:59:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:59:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:59:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:59:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:59:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:59:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 09:59:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:59:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 09:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 09:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 09:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 09:59:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:00:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:00:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:00:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:00:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:00:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:00:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:00:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:00:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:00:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:01:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:01:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:01:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:01:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:01:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:01:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:01:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:01:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:01:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:02:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:02:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:02:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:02:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:02:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:02:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:02:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:02:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:02:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:03:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:03:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:03:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:03:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:03:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:03:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:03:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:03:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:03:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:03:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:04:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:04:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:04:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:04:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:04:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:04:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:04:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:04:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:04:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:05:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:05:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:05:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:05:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:05:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:05:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:05:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:05:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:05:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:06:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:06:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:06:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:06:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:06:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:06:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:06:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:06:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:06:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:06:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:07:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:07:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:07:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:07:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:07:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:07:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:07:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:07:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:07:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:08:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:08:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:08:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:08:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:08:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:08:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:08:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:08:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:08:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:09:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:09:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:09:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:09:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:09:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:09:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:09:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:09:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:09:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:09:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:10:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:10:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:10:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:10:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:10:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:10:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:10:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:10:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:10:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:11:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:11:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:11:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:11:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:11:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:11:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:11:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:11:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:11:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:12:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:12:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:12:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:12:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:12:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:12:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:12:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:12:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:12:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:12:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:13:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:13:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:13:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:13:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:13:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:13:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:13:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:13:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:13:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:14:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:14:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:14:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:14:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:14:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:14:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:14:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:14:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:14:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:15:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:15:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:15:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:15:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:15:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:15:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:15:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:15:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:15:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:15:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:16:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:16:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:16:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:16:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:16:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:16:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:16:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:16:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:16:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:17:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:17:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:17:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:17:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:17:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:17:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:17:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:17:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:17:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:18:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:18:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:18:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:18:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:18:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:18:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:18:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:18:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:18:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:19:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:19:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:19:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:19:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:19:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:19:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:19:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:19:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:19:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:19:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:20:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:20:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:20:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:20:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:20:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:20:32 rdma06 NetworkManager[1025]: <info>  [1680963632.8662] dhcp4 (eno1): state changed new lease, address=10.73.131.45
Apr  8 10:20:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:20:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:20:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:20:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:21:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:21:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:21:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:21:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:21:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:21:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:21:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:21:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:21:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:22:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:22:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:22:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:22:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:22:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:22:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:22:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:22:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:22:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:22:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:23:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:23:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:23:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:23:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:23:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:23:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:23:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:23:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:23:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:24:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:24:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:24:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:24:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:24:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:24:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:24:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:24:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:24:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:25:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:25:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:25:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:25:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:25:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:25:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:25:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:25:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:25:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:25:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:26:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:26:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:26:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:26:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:26:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:26:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:26:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:26:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:26:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:27:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:27:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:27:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:27:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:27:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:27:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:27:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:27:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:27:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:28:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:28:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:28:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:28:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:28:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:28:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:28:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:28:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:28:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:28:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:29:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:29:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:29:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:29:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:29:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:29:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:29:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:29:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:29:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:30:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:30:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:30:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:30:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:30:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:30:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:30:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:30:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:30:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:31:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:31:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:31:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:31:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:31:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:31:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:31:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:31:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:31:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:31:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:32:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:32:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:32:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:32:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:32:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:32:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:32:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:32:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:32:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:33:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:33:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:33:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:33:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:33:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:33:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:33:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:33:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:33:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:34:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:34:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:34:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:34:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:34:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:34:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:34:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:34:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:34:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:35:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:35:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:35:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:35:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:35:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:35:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:35:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:35:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:35:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:35:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:36:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:36:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:36:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:36:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:36:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:36:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:36:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:36:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:36:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:37:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:37:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:37:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:37:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:37:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:37:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:37:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:37:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:37:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:38:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:38:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:38:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:38:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:38:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:38:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:38:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:38:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:38:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:38:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:39:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:39:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:39:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:39:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:39:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:39:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:39:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:39:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:39:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:40:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:40:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:40:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:40:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:40:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:40:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:40:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:40:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:40:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:41:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:41:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:41:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:41:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:41:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:41:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:41:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:41:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:41:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:41:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:42:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:42:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:42:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:42:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:42:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:42:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:42:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:42:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:42:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:43:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:43:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:43:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:43:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:43:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:43:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:43:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:43:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:43:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:44:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:44:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:44:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:44:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:44:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:44:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:44:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:44:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:44:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:44:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:45:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:45:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:45:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:45:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:45:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:45:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:45:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:45:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:45:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:46:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:46:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:46:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:46:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:46:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:46:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:46:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:46:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:46:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:47:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:47:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:47:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:47:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:47:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:47:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:47:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:47:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:47:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:47:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:48:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:48:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:48:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:48:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:48:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:48:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:48:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:48:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:48:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:49:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:49:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:49:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:49:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:49:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:49:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:49:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:49:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:49:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:50:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:50:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:50:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:50:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:50:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:50:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:50:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:50:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:50:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:51:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:51:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:51:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:51:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:51:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:51:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:51:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:51:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:51:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:51:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:52:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:52:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:52:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:52:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:52:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:52:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:52:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:52:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:52:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:53:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:53:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:53:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:53:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:53:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:53:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:53:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:53:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:53:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:54:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:54:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:54:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:54:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:54:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:54:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:54:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:54:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:54:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:54:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:55:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:55:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:55:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:55:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:55:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:55:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:55:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:55:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:55:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:56:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:56:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:56:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:56:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:56:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:56:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:56:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:56:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:56:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:57:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:57:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:57:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:57:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:57:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:57:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:57:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:57:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:57:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:57:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:58:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:58:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:58:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:58:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:58:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:58:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:58:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:58:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:58:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:59:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:59:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:59:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:59:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:59:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:59:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:59:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 10:59:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 10:59:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 10:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 10:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 10:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:00:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:00:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:00:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:00:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:00:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:00:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:00:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:00:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:00:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:00:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:01:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:01:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:01:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:01:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:01:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:01:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:01:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:01:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:01:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:02:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:02:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:02:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:02:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:02:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:02:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:02:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:02:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:02:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:03:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:03:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:03:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:03:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:03:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:03:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:03:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:03:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:03:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:03:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:04:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:04:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:04:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:04:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:04:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:04:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:04:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:04:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:04:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:05:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:05:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:05:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:05:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:05:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:05:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:05:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:05:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:05:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:06:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:06:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:06:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:06:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:06:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:06:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:06:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:06:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:06:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:06:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:07:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:07:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:07:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:07:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:07:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:07:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:07:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:07:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:07:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:08:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:08:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:08:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:08:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:08:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:08:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:08:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:08:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:08:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:09:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:09:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:09:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:09:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:09:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:09:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:09:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:09:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:09:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:10:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:10:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:10:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:10:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:10:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:10:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:10:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:10:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:10:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:10:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:11:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:11:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:11:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:11:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:11:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:11:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:11:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:11:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:11:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:12:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:12:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:12:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:12:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:12:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:12:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:12:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:12:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:12:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:13:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:13:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:13:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:13:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:13:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:13:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:13:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:13:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:13:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:13:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:14:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:14:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:14:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:14:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:14:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:14:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:14:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:14:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:14:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:15:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:15:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:15:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:15:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:15:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:15:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:15:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:15:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:15:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:16:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:16:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:16:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:16:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:16:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:16:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:16:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:16:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:16:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:16:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:17:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:17:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:17:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:17:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:17:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:17:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:17:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:17:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:17:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:18:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:18:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:18:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:18:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:18:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:18:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:18:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:18:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:18:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:19:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:19:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:19:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:19:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:19:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:19:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:19:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:19:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:19:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:19:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:20:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:20:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:20:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:20:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:20:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:20:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:20:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:20:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:20:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:21:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:21:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:21:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:21:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:21:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:21:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:21:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:21:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:21:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:22:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:22:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:22:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:22:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:22:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:22:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:22:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:22:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:22:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:22:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:23:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:23:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:23:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:23:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:23:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:23:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:23:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:23:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:23:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:24:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:24:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:24:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:24:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:24:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:24:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:24:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:24:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:24:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:25:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:25:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:25:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:25:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:25:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:25:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:25:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:25:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:25:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:26:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:26:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:26:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:26:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:26:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:26:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:26:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:26:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:26:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:26:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:27:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:27:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:27:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:27:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:27:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:27:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:27:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:27:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:27:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:28:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:28:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:28:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:28:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:28:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:28:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:28:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:28:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:28:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:29:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:29:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:29:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:29:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:29:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:29:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:29:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:29:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:29:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:29:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:30:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  8 11:30:01 rdma06 dnf[634523]: Updating Subscription Management repositories.
Apr  8 11:30:01 rdma06 dnf[634523]: Unable to read consumer identity
Apr  8 11:30:01 rdma06 dnf[634523]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  8 11:30:01 rdma06 dnf[634523]: Metadata cache refreshed recently.
Apr  8 11:30:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  8 11:30:01 rdma06 systemd[1]: Finished dnf makecache.
Apr  8 11:30:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:30:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:30:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:30:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:30:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:30:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:30:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:30:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:30:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:31:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:31:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:31:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:31:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:31:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:31:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:31:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:31:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:31:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:32:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:32:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:32:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:32:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:32:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:32:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:32:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:32:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:32:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:32:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:33:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:33:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:33:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:33:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:33:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:33:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:33:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:33:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:33:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:34:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:34:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:34:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:34:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:34:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:34:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:34:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:34:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:34:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:35:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:35:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:35:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:35:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:35:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:35:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:35:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:35:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:35:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:35:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:36:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:36:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:36:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:36:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:36:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:36:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:36:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:36:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:36:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:37:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:37:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:37:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:37:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:37:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:37:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:37:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:37:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:37:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:38:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:38:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:38:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:38:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:38:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:38:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:38:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:38:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:38:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:38:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:39:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:39:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:39:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:39:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:39:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:39:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:39:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:39:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:39:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:40:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:40:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:40:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:40:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:40:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:40:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:40:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:40:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:40:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:41:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:41:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:41:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:41:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:41:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:41:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:41:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:41:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:41:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:42:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:42:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:42:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:42:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:42:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:42:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:42:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:42:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:42:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:42:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:43:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:43:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:43:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:43:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:43:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:43:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:43:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:43:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:43:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:44:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:44:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:44:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:44:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:44:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:44:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:44:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:44:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:44:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:45:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:45:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:45:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:45:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:45:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:45:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:45:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:45:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:45:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:45:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:46:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:46:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:46:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:46:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:46:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:46:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:46:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:46:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:46:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:47:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:47:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:47:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:47:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:47:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:47:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:47:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:47:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:47:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:48:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:48:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:48:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:48:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:48:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:48:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:48:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:48:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:48:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:48:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:49:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:49:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:49:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:49:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:49:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:49:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:49:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:49:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:49:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:50:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:50:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:50:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:50:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:50:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:50:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:50:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:50:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:50:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:51:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:51:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:51:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:51:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:51:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:51:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:51:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:51:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:51:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:51:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:52:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:52:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:52:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:52:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:52:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:52:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:52:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:52:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:52:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:53:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:53:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:53:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:53:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:53:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:53:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:53:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:53:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:53:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:54:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:54:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:54:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:54:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:54:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:54:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:54:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:54:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:54:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:54:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:55:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:55:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:55:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:55:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:55:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:55:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:55:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:55:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:55:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:56:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:56:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:56:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:56:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:56:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:56:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:56:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:56:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:56:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:57:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:57:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:57:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:57:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:57:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:57:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:57:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:57:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:57:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:58:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:58:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:58:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:58:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:58:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:58:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:58:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:58:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:58:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:58:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:59:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:59:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:59:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:59:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:59:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:59:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:59:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 11:59:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 11:59:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 11:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 11:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 11:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:00:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:00:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:00:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:00:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:00:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:00:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:00:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:00:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:00:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:01:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:01:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:01:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:01:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:01:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:01:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:01:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:01:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:01:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:01:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:02:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:02:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:02:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:02:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:02:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:02:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:02:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:02:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:02:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:03:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:03:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:03:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:03:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:03:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:03:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:03:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:03:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:03:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:04:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:04:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:04:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:04:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:04:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:04:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:04:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:04:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:04:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:04:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:05:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:05:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:05:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:05:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:05:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:05:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:05:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:05:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:05:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:06:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:06:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:06:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:06:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:06:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:06:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:06:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:06:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:06:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:07:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:07:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:07:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:07:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:07:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:07:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:07:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:07:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:07:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:07:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:08:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:08:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:08:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:08:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:08:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:08:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:08:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:08:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:08:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:09:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:09:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:09:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:09:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:09:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:09:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:09:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:09:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:09:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:10:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:10:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:10:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:10:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:10:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:10:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:10:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:10:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:10:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:10:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:11:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:11:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:11:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:11:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:11:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:11:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:11:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:11:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:11:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:12:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:12:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:12:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:12:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:12:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:12:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:12:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:12:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:12:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:13:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:13:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:13:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:13:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:13:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:13:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:13:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:13:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:13:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:13:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:14:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:14:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:14:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:14:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:14:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:14:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:14:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:14:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:14:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:15:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:15:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:15:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:15:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:15:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:15:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:15:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:15:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:15:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:16:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:16:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:16:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:16:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:16:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:16:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:16:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:16:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:16:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:17:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:17:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:17:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:17:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:17:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:17:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:17:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:17:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:17:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:17:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:18:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:18:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:18:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:18:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:18:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:18:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:18:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:18:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:18:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:19:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:19:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:19:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:19:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:19:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:19:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:19:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:19:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:19:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:20:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:20:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:20:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:20:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:20:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:20:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:20:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:20:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:20:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:20:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:21:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:21:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:21:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:21:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:21:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:21:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:21:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:21:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:21:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:22:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:22:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:22:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:22:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:22:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:22:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:22:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:22:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:22:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:23:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:23:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:23:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:23:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:23:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:23:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:23:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:23:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:23:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:23:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:24:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:24:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:24:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:24:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:24:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:24:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:24:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:24:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:24:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:25:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:25:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:25:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:25:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:25:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:25:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:25:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:25:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:25:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:26:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:26:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:26:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:26:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:26:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:26:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:26:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:26:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:26:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:26:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:27:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:27:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:27:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:27:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:27:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:27:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:27:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:27:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:27:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:28:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:28:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:28:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:28:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:28:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:28:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:28:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:28:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:28:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:29:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:29:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:29:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:29:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:29:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:29:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:29:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:29:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:29:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:29:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:30:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:30:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:30:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:30:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:30:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:30:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:30:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:30:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:30:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:31:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:31:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:31:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:31:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:31:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:31:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:31:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:31:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:31:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:32:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:32:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:32:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:32:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:32:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:32:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:32:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:32:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:32:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:33:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:33:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:33:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:33:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:33:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:33:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:33:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:33:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:33:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:33:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:34:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:34:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:34:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:34:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:34:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:34:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:34:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:34:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:34:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:35:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:35:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:35:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:35:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:35:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:35:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:35:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:35:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:35:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:36:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:36:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:36:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:36:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:36:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:36:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:36:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:36:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:36:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:36:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:37:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:37:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:37:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:37:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:37:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:37:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:37:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:37:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:37:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:38:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:38:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:38:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:38:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:38:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:38:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:38:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:38:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:38:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:39:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:39:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:39:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:39:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:39:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:39:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:39:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:39:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:39:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:39:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:40:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:40:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:40:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:40:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:40:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:40:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:40:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:40:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:40:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:41:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:41:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:41:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:41:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:41:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:41:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:41:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:41:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:41:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:42:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:42:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:42:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:42:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:42:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:42:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:42:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:42:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:42:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:42:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:43:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:43:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:43:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:43:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:43:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:43:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:43:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:43:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:43:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:44:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:44:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:44:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:44:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:44:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:44:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:44:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:44:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:44:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:45:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:45:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:45:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:45:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:45:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:45:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:45:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:45:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:45:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:45:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:46:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:46:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:46:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:46:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:46:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:46:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:46:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:46:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:46:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:47:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:47:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:47:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:47:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:47:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:47:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:47:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:47:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:47:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:48:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:48:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:48:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:48:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:48:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:48:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:48:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:48:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:48:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:49:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:49:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:49:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:49:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:49:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:49:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:49:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:49:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:49:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:49:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:50:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:50:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:50:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:50:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:50:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:50:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:50:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:50:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:50:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:51:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:51:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:51:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:51:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:51:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:51:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:51:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:51:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:51:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:52:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:52:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:52:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:52:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:52:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:52:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:52:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:52:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:52:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:52:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:53:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:53:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:53:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:53:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:53:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:53:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:53:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:53:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:53:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:54:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:54:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:54:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:54:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:54:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:54:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:54:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:54:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:54:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:55:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:55:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:55:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:55:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:55:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:55:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:55:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:55:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:55:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:55:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:56:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:56:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:56:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:56:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:56:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:56:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:56:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:56:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:56:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:57:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:57:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:57:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:57:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:57:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:57:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:57:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:57:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:57:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:58:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:58:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:58:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:58:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:58:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:58:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:58:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:58:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:58:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:58:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:59:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:59:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:59:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:59:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:59:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:59:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:59:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 12:59:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 12:59:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 12:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 12:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 12:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:00:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:00:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:00:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:00:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:00:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:00:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:00:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:00:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:00:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:01:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:01:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:01:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:01:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:01:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:01:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:01:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:01:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:01:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:01:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:02:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:02:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:02:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:02:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:02:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:02:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:02:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:02:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:02:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:03:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:03:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:03:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:03:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:03:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:03:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:03:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:03:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:03:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:04:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:04:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:04:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:04:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:04:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:04:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:04:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:04:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:04:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:05:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:05:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:05:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:05:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:05:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:05:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:05:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:05:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:05:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:05:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:06:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:06:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:06:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:06:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:06:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:06:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:06:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:06:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:06:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:07:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:07:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:07:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:07:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:07:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:07:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:07:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:07:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:07:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:08:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:08:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:08:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:08:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:08:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:08:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:08:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:08:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:08:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:08:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:09:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:09:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:09:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:09:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:09:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:09:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:09:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:09:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:09:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:10:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:10:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:10:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:10:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:10:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:10:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:10:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:10:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:10:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:11:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:11:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:11:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:11:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:11:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:11:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:11:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:11:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:11:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:11:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:12:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:12:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:12:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:12:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:12:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:12:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:12:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:12:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:12:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:13:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:13:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:13:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:13:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:13:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:13:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:13:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:13:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:13:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:14:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:14:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:14:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:14:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:14:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:14:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:14:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:14:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:14:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:14:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:15:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:15:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:15:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:15:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:15:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:15:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:15:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:15:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:15:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:16:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:16:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:16:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:16:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:16:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:16:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:16:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:16:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:16:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:17:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:17:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:17:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:17:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:17:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:17:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:17:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:17:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:17:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:17:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:18:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:18:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:18:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:18:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:18:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:18:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:18:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:18:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:18:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:19:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:19:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:19:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:19:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:19:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:19:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:19:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:19:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:19:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:20:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:20:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:20:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:20:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:20:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:20:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:20:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:20:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:20:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:20:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:21:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:21:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:21:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:21:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:21:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:21:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:21:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:21:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:21:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:22:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:22:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:22:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:22:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:22:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:22:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:22:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:22:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:22:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:23:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  8 13:23:01 rdma06 dnf[639098]: Updating Subscription Management repositories.
Apr  8 13:23:01 rdma06 dnf[639098]: Unable to read consumer identity
Apr  8 13:23:01 rdma06 dnf[639098]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  8 13:23:02 rdma06 dnf[639098]: Copr repo for fmf owned by psss                 2.4 kB/s | 3.3 kB     00:01
Apr  8 13:23:03 rdma06 dnf[639098]: beaker-AppStream                                 96 kB/s | 3.2 kB     00:00
Apr  8 13:23:03 rdma06 dnf[639098]: beaker-AppStream-debuginfo                       41 kB/s | 1.5 kB     00:00
Apr  8 13:23:03 rdma06 dnf[639098]: beaker-BaseOS                                   147 kB/s | 2.7 kB     00:00
Apr  8 13:23:03 rdma06 dnf[639098]: beaker-BaseOS-debuginfo                          49 kB/s | 1.5 kB     00:00
Apr  8 13:23:03 rdma06 dnf[639098]: beaker-CRB                                       72 kB/s | 2.7 kB     00:00
Apr  8 13:23:03 rdma06 dnf[639098]: beaker-CRB-debuginfo                             63 kB/s | 1.5 kB     00:00
Apr  8 13:23:03 rdma06 dnf[639098]: beaker-HighAvailability                          80 kB/s | 2.8 kB     00:00
Apr  8 13:23:03 rdma06 dnf[639098]: beaker-HighAvailability-debuginfo                36 kB/s | 1.5 kB     00:00
Apr  8 13:23:03 rdma06 dnf[639098]: beaker-NFV                                       91 kB/s | 2.7 kB     00:00
Apr  8 13:23:03 rdma06 dnf[639098]: beaker-NFV-debuginfo                             69 kB/s | 1.5 kB     00:00
Apr  8 13:23:03 rdma06 dnf[639098]: beaker-RT                                        37 kB/s | 2.7 kB     00:00
Apr  8 13:23:03 rdma06 dnf[639098]: beaker-RT-debuginfo                              79 kB/s | 1.5 kB     00:00
Apr  8 13:23:03 rdma06 dnf[639098]: beaker-ResilientStorage                          65 kB/s | 2.8 kB     00:00
Apr  8 13:23:03 rdma06 dnf[639098]: beaker-ResilientStorage-debuginfo                60 kB/s | 1.5 kB     00:00
Apr  8 13:23:03 rdma06 dnf[639098]: beaker-SAP                                      126 kB/s | 2.7 kB     00:00
Apr  8 13:23:03 rdma06 dnf[639098]: beaker-SAP-debuginfo                             59 kB/s | 1.5 kB     00:00
Apr  8 13:23:03 rdma06 dnf[639098]: beaker-SAPHANA                                   95 kB/s | 2.7 kB     00:00
Apr  8 13:23:03 rdma06 dnf[639098]: beaker-SAPHANA-debuginfo                         48 kB/s | 1.5 kB     00:00
Apr  8 13:23:04 rdma06 dnf[639098]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr  8 13:23:05 rdma06 dnf[639098]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr  8 13:23:05 rdma06 dnf[639098]: Metadata cache created.
Apr  8 13:23:05 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  8 13:23:05 rdma06 systemd[1]: Finished dnf makecache.
Apr  8 13:23:05 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.428s CPU time.
Apr  8 13:23:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:23:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:23:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:23:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:23:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:23:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:23:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:23:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:23:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:24:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:24:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:24:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:24:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:24:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:24:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:24:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:24:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:24:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:24:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:25:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:25:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:25:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:25:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:25:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:25:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:25:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:25:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:25:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:26:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:26:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:26:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:26:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:26:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:26:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:26:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:26:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:26:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:27:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:27:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:27:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:27:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:27:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:27:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:27:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:27:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:27:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:27:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:28:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:28:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:28:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:28:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:28:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:28:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:28:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:28:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:28:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:29:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:29:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:29:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:29:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:29:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:29:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:29:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:29:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:29:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:30:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:30:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:30:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:30:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:30:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:30:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:30:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:30:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:30:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:30:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:31:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:31:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:31:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:31:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:31:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:31:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:31:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:31:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:31:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:32:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:32:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:32:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:32:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:32:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:32:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:32:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:32:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:32:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:33:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:33:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:33:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:33:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:33:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:33:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:33:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:33:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:33:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:33:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:34:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:34:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:34:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:34:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:34:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:34:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:34:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:34:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:34:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:35:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:35:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:35:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:35:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:35:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:35:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:35:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:35:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:35:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:36:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:36:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:36:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:36:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:36:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:36:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:36:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:36:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:36:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:36:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:37:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:37:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:37:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:37:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:37:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:37:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:37:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:37:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:37:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:38:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:38:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:38:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:38:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:38:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:38:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:38:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:38:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:38:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:39:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:39:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:39:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:39:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:39:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:39:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:39:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:39:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:39:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:40:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:40:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:40:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:40:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:40:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:40:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:40:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:40:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:40:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:40:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:41:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:41:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:41:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:41:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:41:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:41:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:41:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:41:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:41:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:42:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:42:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:42:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:42:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:42:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:42:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:42:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:42:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:42:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:43:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:43:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:43:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:43:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:43:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:43:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:43:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:43:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:43:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:43:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:44:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:44:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:44:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:44:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:44:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:44:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:44:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:44:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:44:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:45:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:45:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:45:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:45:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:45:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:45:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:45:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:45:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:45:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:46:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:46:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:46:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:46:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:46:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:46:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:46:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:46:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:46:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:46:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:47:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:47:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:47:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:47:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:47:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:47:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:47:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:47:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:47:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:48:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:48:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:48:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:48:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:48:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:48:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:48:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:48:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:48:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:49:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:49:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:49:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:49:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:49:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:49:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:49:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:49:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:49:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:49:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:50:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:50:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:50:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:50:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:50:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:50:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:50:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:50:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:50:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:51:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:51:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:51:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:51:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:51:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:51:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:51:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:51:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:51:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:52:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:52:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:52:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:52:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:52:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:52:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:52:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:52:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:52:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:52:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:53:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:53:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:53:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:53:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:53:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:53:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:53:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:53:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:53:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:54:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:54:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:54:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:54:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:54:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:54:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:54:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:54:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:54:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:55:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:55:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:55:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:55:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:55:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:55:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:55:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:55:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:55:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:56:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:56:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:56:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:56:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:56:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:56:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:56:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:56:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:56:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:56:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:57:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:57:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:57:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:57:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:57:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:57:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:57:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:57:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:57:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:58:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:58:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:58:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:58:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:58:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:58:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:58:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:58:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:58:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:59:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:59:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:59:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:59:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:59:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:59:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:59:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:59:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 13:59:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 13:59:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 13:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 13:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 13:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:00:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:00:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:00:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:00:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:00:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:00:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:00:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:00:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:00:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:01:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:01:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:01:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:01:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:01:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:01:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:01:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:01:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:01:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:02:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:02:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:02:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:02:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:02:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:02:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:02:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:02:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:02:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:02:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:03:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:03:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:03:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:03:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:03:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:03:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:03:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:03:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:03:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:04:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:04:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:04:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:04:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:04:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:04:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:04:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:04:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:04:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:05:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:05:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:05:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:05:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:05:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:05:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:05:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:05:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:05:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:05:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:06:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:06:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:06:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:06:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:06:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:06:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:06:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:06:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:06:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:07:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:07:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:07:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:07:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:07:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:07:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:07:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:07:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:07:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:08:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:08:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:08:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:08:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:08:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:08:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:08:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:08:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:08:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:08:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:09:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:09:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:09:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:09:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:09:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:09:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:09:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:09:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:09:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:10:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:10:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:10:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:10:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:10:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:10:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:10:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:10:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:10:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:11:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:11:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:11:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:11:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:11:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:11:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:11:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:11:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:11:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:12:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:12:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:12:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:12:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:12:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:12:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:12:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:12:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:12:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:12:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:13:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:13:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:13:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:13:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:13:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:13:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:13:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:13:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:13:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:14:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:14:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:14:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:14:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:14:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:14:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:14:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:14:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:14:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:15:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:15:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:15:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:15:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:15:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:15:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:15:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:15:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:15:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:15:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:16:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:16:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:16:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:16:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:16:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:16:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:16:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:16:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:16:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:17:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:17:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:17:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:17:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:17:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:17:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:17:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:17:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:17:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:18:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:18:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:18:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:18:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:18:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:18:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:18:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:18:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:18:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:18:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:19:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:19:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:19:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:19:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:19:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:19:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:19:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:19:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:19:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:20:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:20:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:20:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:20:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:20:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:20:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:20:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:20:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:20:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:21:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:21:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:21:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:21:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:21:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:21:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:21:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:21:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:21:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:21:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:22:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:22:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:22:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:22:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:22:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:22:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:22:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:22:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:22:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:23:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:23:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:23:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:23:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:23:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:23:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:23:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:23:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:23:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:24:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:24:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:24:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:24:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:24:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:24:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:24:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:24:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:24:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:24:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:25:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:25:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:25:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:25:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:25:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:25:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:25:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:25:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:25:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:26:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:26:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:26:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:26:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:26:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:26:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:26:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:26:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:26:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:27:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:27:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:27:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:27:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:27:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:27:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:27:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:27:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:27:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:27:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:28:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:28:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:28:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:28:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:28:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:28:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:28:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:28:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:28:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:29:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:29:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:29:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:29:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:29:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:29:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:29:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:29:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:29:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:30:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:30:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:30:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:30:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:30:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:30:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:30:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:30:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:30:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:31:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:31:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:31:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:31:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:31:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:31:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:31:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:31:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:31:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:31:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:32:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:32:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:32:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:32:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:32:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:32:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:32:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:32:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:32:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:33:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:33:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:33:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:33:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:33:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:33:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:33:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:33:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:33:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:34:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:34:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:34:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:34:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:34:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:34:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:34:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:34:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:34:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:34:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:35:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:35:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:35:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:35:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:35:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:35:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:35:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:35:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:35:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:36:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:36:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:36:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:36:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:36:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:36:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:36:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:36:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:36:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:37:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:37:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:37:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:37:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:37:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:37:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:37:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:37:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:37:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:37:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:38:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:38:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:38:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:38:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:38:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:38:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:38:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:38:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:38:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:39:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:39:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:39:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:39:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:39:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:39:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:39:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:39:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:39:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:40:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:40:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:40:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:40:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:40:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:40:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:40:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:40:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:40:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:40:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:41:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:41:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:41:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:41:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:41:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:41:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:41:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:41:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:41:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:42:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:42:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:42:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:42:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:42:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:42:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:42:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:42:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:42:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:43:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:43:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:43:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:43:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:43:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:43:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:43:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:43:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:43:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:43:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:44:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  8 14:44:01 rdma06 dnf[642394]: Updating Subscription Management repositories.
Apr  8 14:44:01 rdma06 dnf[642394]: Unable to read consumer identity
Apr  8 14:44:01 rdma06 dnf[642394]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  8 14:44:01 rdma06 dnf[642394]: Metadata cache refreshed recently.
Apr  8 14:44:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  8 14:44:01 rdma06 systemd[1]: Finished dnf makecache.
Apr  8 14:44:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:44:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:44:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:44:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:44:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:44:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:44:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:44:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:44:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:45:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:45:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:45:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:45:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:45:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:45:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:45:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:45:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:45:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:46:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:46:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:46:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:46:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:46:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:46:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:46:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:46:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:46:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:47:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:47:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:47:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:47:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:47:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:47:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:47:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:47:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:47:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:47:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:48:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:48:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:48:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:48:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:48:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:48:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:48:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:48:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:48:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:49:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:49:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:49:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:49:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:49:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:49:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:49:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:49:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:49:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:50:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:50:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:50:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:50:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:50:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:50:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:50:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:50:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:50:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:50:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:51:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:51:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:51:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:51:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:51:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:51:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:51:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:51:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:51:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:52:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:52:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:52:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:52:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:52:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:52:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:52:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:52:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:52:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:53:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:53:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:53:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:53:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:53:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:53:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:53:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:53:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:53:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:53:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:54:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:54:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:54:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:54:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:54:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:54:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:54:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:54:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:54:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:55:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:55:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:55:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:55:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:55:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:55:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:55:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:55:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:55:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:56:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:56:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:56:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:56:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:56:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:56:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:56:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:56:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:56:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:56:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:57:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:57:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:57:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:57:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:57:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:57:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:57:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:57:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:57:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:58:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:58:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:58:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:58:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:58:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:58:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:58:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:58:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:58:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:59:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:59:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:59:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:59:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:59:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:59:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:59:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 14:59:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:59:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 14:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 14:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 14:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 14:59:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:00:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:00:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:00:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:00:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:00:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:00:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:00:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:00:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:00:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:01:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:01:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:01:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:01:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:01:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:01:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:01:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:01:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:01:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:02:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:02:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:02:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:02:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:02:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:02:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:02:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:02:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:02:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:03:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:03:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:03:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:03:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:03:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:03:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:03:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:03:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:03:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:03:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:04:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:04:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:04:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:04:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:04:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:04:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:04:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:04:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:04:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:05:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:05:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:05:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:05:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:05:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:05:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:05:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:05:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:05:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:06:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:06:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:06:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:06:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:06:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:06:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:06:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:06:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:06:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:06:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:07:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:07:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:07:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:07:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:07:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:07:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:07:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:07:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:07:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:08:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:08:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:08:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:08:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:08:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:08:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:08:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:08:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:08:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:09:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:09:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:09:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:09:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:09:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:09:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:09:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:09:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:09:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:09:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:10:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:10:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:10:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:10:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:10:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:10:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:10:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:10:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:10:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:11:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:11:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:11:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:11:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:11:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:11:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:11:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:11:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:11:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:12:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:12:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:12:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:12:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:12:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:12:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:12:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:12:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:12:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:12:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:13:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:13:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:13:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:13:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:13:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:13:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:13:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:13:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:13:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:14:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:14:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:14:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:14:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:14:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:14:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:14:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:14:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:14:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:15:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:15:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:15:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:15:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:15:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:15:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:15:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:15:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:15:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:15:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:16:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:16:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:16:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:16:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:16:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:16:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:16:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:16:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:16:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:17:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:17:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:17:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:17:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:17:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:17:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:17:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:17:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:17:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:18:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:18:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:18:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:18:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:18:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:18:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:18:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:18:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:18:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:19:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:19:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:19:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:19:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:19:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:19:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:19:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:19:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:19:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:19:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:20:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:20:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:20:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:20:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:20:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:20:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:20:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:20:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:20:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:21:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:21:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:21:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:21:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:21:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:21:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:21:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:21:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:21:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:22:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:22:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:22:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:22:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:22:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:22:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:22:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:22:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:22:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:22:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:23:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:23:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:23:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:23:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:23:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:23:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:23:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:23:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:23:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:24:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:24:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:24:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:24:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:24:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:24:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:24:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:24:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:24:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:25:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:25:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:25:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:25:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:25:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:25:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:25:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:25:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:25:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:25:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:26:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:26:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:26:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:26:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:26:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:26:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:26:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:26:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:26:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:27:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:27:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:27:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:27:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:27:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:27:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:27:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:27:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:27:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:28:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:28:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:28:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:28:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:28:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:28:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:28:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:28:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:28:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:28:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:29:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:29:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:29:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:29:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:29:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:29:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:29:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:29:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:29:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:30:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:30:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:30:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:30:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:30:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:30:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:30:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:30:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:30:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:31:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:31:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:31:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:31:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:31:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:31:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:31:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:31:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:31:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:31:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:32:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:32:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:32:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:32:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:32:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:32:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:32:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:32:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:32:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:33:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:33:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:33:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:33:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:33:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:33:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:33:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:33:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:33:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:34:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:34:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:34:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:34:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:34:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:34:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:34:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:34:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:34:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:34:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:35:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:35:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:35:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:35:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:35:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:35:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:35:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:35:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:35:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:36:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:36:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:36:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:36:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:36:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:36:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:36:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:36:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:36:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:37:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:37:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:37:12 rdma06 chronyd[1034]: Selected source 10.11.160.238
Apr  8 15:37:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:37:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:37:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:37:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:37:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:37:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:37:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:38:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:38:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:38:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:38:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:38:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:38:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:38:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:38:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:38:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:38:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:39:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:39:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:39:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:39:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:39:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:39:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:39:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:39:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:39:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:40:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:40:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:40:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:40:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:40:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:40:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:40:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:40:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:40:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:41:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:41:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:41:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:41:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:41:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:41:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:41:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:41:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:41:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:41:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:42:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:42:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:42:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:42:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:42:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:42:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:42:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:42:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:42:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:43:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:43:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:43:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:43:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:43:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:43:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:43:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:43:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:43:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:44:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:44:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:44:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:44:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:44:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:44:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:44:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:44:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:44:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:44:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:45:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:45:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:45:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:45:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:45:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:45:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:45:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:45:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:45:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:46:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:46:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:46:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:46:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:46:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:46:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:46:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:46:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:46:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:47:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:47:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:47:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:47:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:47:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:47:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:47:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:47:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:47:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:47:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:48:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:48:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:48:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:48:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:48:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:48:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:48:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:48:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:48:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:49:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:49:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:49:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:49:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:49:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:49:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:49:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:49:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:49:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:50:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:50:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:50:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:50:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:50:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:50:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:50:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:50:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:50:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:50:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:51:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:51:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:51:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:51:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:51:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:51:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:51:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:51:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:51:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:52:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:52:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:52:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:52:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:52:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:52:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:52:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:52:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:52:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:53:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:53:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:53:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:53:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:53:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:53:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:53:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:53:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:53:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:54:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:54:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:54:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:54:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:54:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:54:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:54:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:54:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:54:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:54:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:55:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:55:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:55:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:55:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:55:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:55:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:55:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:55:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:55:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:56:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:56:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:56:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:56:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:56:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:56:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:56:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:56:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:56:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:57:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:57:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:57:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:57:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:57:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:57:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:57:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:57:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:57:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:57:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:58:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:58:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:58:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:58:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:58:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:58:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:58:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:58:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:58:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:59:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:59:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:59:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:59:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:59:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:59:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:59:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 15:59:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 15:59:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 15:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 15:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 15:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:00:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:00:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:00:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:00:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:00:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:00:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:00:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:00:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:00:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:00:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:01:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:01:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:01:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:01:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:01:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:01:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:01:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:01:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:01:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:02:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:02:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:02:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:02:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:02:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:02:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:02:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:02:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:02:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:03:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:03:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:03:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:03:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:03:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:03:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:03:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:03:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:03:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:03:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:04:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:04:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:04:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:04:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:04:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:04:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:04:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:04:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:04:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:05:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:05:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:05:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:05:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:05:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:05:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:05:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:05:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:05:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:06:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:06:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:06:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:06:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:06:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:06:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:06:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:06:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:06:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:06:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:07:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:07:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:07:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:07:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:07:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:07:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:07:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:07:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:07:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:08:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:08:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:08:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:08:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:08:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:08:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:08:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:08:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:08:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:09:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:09:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:09:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:09:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:09:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:09:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:09:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:09:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:09:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:10:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:10:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:10:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:10:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:10:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:10:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:10:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:10:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:10:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:10:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:11:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:11:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:11:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:11:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:11:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:11:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:11:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:11:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:11:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:12:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:12:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:12:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:12:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:12:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:12:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:12:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:12:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:12:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:13:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:13:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:13:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:13:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:13:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:13:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:13:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:13:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:13:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:13:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:14:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:14:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:14:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:14:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:14:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:14:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:14:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:14:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:14:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:15:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:15:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:15:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:15:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:15:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:15:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:15:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:15:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:15:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:16:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:16:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:16:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:16:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:16:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:16:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:16:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:16:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:16:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:16:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:17:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:17:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:17:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:17:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:17:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:17:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:17:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:17:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:17:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:18:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:18:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:18:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:18:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:18:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:18:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:18:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:18:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:18:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:19:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:19:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:19:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:19:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:19:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:19:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:19:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:19:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:19:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:19:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:20:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:20:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:20:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:20:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:20:32 rdma06 NetworkManager[1025]: <info>  [1680985232.9222] dhcp4 (eno1): state changed new lease, address=10.73.131.45
Apr  8 16:20:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:20:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:20:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:20:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:20:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:21:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:21:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:21:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:21:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:21:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:21:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:21:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:21:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:21:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:22:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:22:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:22:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:22:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:22:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:22:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:22:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:22:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:22:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:22:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:23:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:23:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:23:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:23:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:23:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:23:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:23:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:23:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:23:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:24:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:24:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:24:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:24:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:24:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:24:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:24:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:24:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:24:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:25:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:25:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:25:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:25:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:25:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:25:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:25:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:25:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:25:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:26:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:26:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:26:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:26:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:26:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:26:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:26:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:26:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:26:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:26:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:27:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:27:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:27:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:27:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:27:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:27:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:27:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:27:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:27:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:28:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:28:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:28:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:28:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:28:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:28:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:28:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:28:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:28:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:29:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:29:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:29:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:29:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:29:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:29:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:29:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:29:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:29:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:29:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:30:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:30:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:30:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:30:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:30:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:30:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:30:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:30:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:30:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:31:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:31:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:31:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:31:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:31:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:31:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:31:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:31:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:31:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:32:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:32:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:32:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:32:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:32:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:32:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:32:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:32:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:32:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:32:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:33:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:33:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:33:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:33:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:33:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:33:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:33:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:33:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:33:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:34:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:34:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:34:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:34:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:34:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:34:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:34:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:34:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:34:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:35:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:35:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:35:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:35:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:35:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:35:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:35:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:35:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:35:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:35:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:36:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:36:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:36:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:36:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:36:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:36:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:36:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:36:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:36:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:37:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:37:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:37:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:37:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:37:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:37:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:37:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:37:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:37:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:38:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:38:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:38:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:38:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:38:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:38:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:38:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:38:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:38:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:38:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:39:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  8 16:39:01 rdma06 dnf[647043]: Updating Subscription Management repositories.
Apr  8 16:39:01 rdma06 dnf[647043]: Unable to read consumer identity
Apr  8 16:39:01 rdma06 dnf[647043]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  8 16:39:03 rdma06 dnf[647043]: Copr repo for fmf owned by psss                 1.9 kB/s | 3.3 kB     00:01
Apr  8 16:39:04 rdma06 dnf[647043]: beaker-AppStream                                2.7 kB/s | 3.2 kB     00:01
Apr  8 16:39:04 rdma06 dnf[647043]: beaker-AppStream-debuginfo                       41 kB/s | 1.5 kB     00:00
Apr  8 16:39:04 rdma06 dnf[647043]: beaker-BaseOS                                    65 kB/s | 2.7 kB     00:00
Apr  8 16:39:04 rdma06 dnf[647043]: beaker-BaseOS-debuginfo                          40 kB/s | 1.5 kB     00:00
Apr  8 16:39:04 rdma06 dnf[647043]: beaker-CRB                                       50 kB/s | 2.7 kB     00:00
Apr  8 16:39:04 rdma06 dnf[647043]: beaker-CRB-debuginfo                             64 kB/s | 1.5 kB     00:00
Apr  8 16:39:04 rdma06 dnf[647043]: beaker-HighAvailability                          79 kB/s | 2.8 kB     00:00
Apr  8 16:39:05 rdma06 dnf[647043]: beaker-HighAvailability-debuginfo                50 kB/s | 1.5 kB     00:00
Apr  8 16:39:05 rdma06 dnf[647043]: beaker-NFV                                       77 kB/s | 2.7 kB     00:00
Apr  8 16:39:05 rdma06 dnf[647043]: beaker-NFV-debuginfo                             57 kB/s | 1.5 kB     00:00
Apr  8 16:39:05 rdma06 dnf[647043]: beaker-RT                                        54 kB/s | 2.7 kB     00:00
Apr  8 16:39:05 rdma06 dnf[647043]: beaker-RT-debuginfo                              41 kB/s | 1.5 kB     00:00
Apr  8 16:39:05 rdma06 dnf[647043]: beaker-ResilientStorage                          60 kB/s | 2.8 kB     00:00
Apr  8 16:39:05 rdma06 dnf[647043]: beaker-ResilientStorage-debuginfo                56 kB/s | 1.5 kB     00:00
Apr  8 16:39:05 rdma06 dnf[647043]: beaker-SAP                                      139 kB/s | 2.7 kB     00:00
Apr  8 16:39:05 rdma06 dnf[647043]: beaker-SAP-debuginfo                             59 kB/s | 1.5 kB     00:00
Apr  8 16:39:05 rdma06 dnf[647043]: beaker-SAPHANA                                   51 kB/s | 2.7 kB     00:00
Apr  8 16:39:05 rdma06 dnf[647043]: beaker-SAPHANA-debuginfo                         49 kB/s | 1.5 kB     00:00
Apr  8 16:39:06 rdma06 dnf[647043]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr  8 16:39:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:39:06 rdma06 dnf[647043]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr  8 16:39:07 rdma06 dnf[647043]: Metadata cache created.
Apr  8 16:39:07 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  8 16:39:07 rdma06 systemd[1]: Finished dnf makecache.
Apr  8 16:39:07 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.418s CPU time.
Apr  8 16:39:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:39:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:39:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:39:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:39:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:39:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:39:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:39:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:40:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:40:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:40:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:40:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:40:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:40:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:40:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:40:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:40:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:41:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:41:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:41:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:41:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:41:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:41:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:41:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:41:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:41:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:41:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:42:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:42:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:42:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:42:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:42:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:42:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:42:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:42:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:42:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:43:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:43:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:43:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:43:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:43:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:43:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:43:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:43:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:43:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:44:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:44:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:44:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:44:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:44:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:44:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:44:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:44:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:44:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:45:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:45:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:45:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:45:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:45:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:45:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:45:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:45:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:45:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:45:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:46:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:46:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:46:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:46:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:46:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:46:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:46:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:46:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:46:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:47:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:47:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:47:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:47:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:47:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:47:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:47:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:47:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:47:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:48:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:48:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:48:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:48:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:48:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:48:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:48:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:48:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:48:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:48:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:49:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:49:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:49:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:49:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:49:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:49:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:49:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:49:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:49:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:50:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:50:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:50:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:50:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:50:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:50:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:50:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:50:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:50:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:51:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:51:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:51:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:51:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:51:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:51:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:51:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:51:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:51:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:51:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:52:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:52:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:52:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:52:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:52:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:52:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:52:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:52:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:52:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:53:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:53:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:53:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:53:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:53:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:53:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:53:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:53:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:53:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:54:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:54:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:54:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:54:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:54:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:54:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:54:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:54:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:54:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:54:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:55:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:55:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:55:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:55:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:55:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:55:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:55:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:55:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:55:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:56:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:56:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:56:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:56:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:56:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:56:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:56:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:56:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:56:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:57:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:57:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:57:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:57:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:57:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:57:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:57:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:57:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:57:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:57:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:58:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:58:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:58:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:58:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:58:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:58:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:58:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:58:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:58:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:59:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:59:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:59:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:59:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:59:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:59:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:59:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 16:59:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 16:59:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 16:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 16:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 16:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:00:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:00:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:00:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:00:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:00:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:00:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:00:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:00:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:00:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:01:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:01:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:01:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:01:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:01:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:01:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:01:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:01:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:01:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:01:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:02:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:02:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:02:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:02:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:02:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:02:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:02:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:02:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:02:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:03:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:03:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:03:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:03:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:03:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:03:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:03:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:03:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:03:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:04:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:04:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:04:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:04:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:04:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:04:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:04:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:04:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:04:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:04:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:05:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:05:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:05:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:05:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:05:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:05:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:05:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:05:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:05:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:06:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:06:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:06:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:06:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:06:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:06:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:06:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:06:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:06:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:07:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:07:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:07:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:07:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:07:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:07:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:07:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:07:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:07:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:07:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:08:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:08:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:08:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:08:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:08:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:08:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:08:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:08:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:08:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:09:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:09:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:09:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:09:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:09:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:09:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:09:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:09:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:09:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:10:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:10:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:10:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:10:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:10:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:10:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:10:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:10:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:10:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:10:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:11:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:11:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:11:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:11:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:11:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:11:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:11:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:11:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:11:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:12:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:12:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:12:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:12:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:12:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:12:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:12:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:12:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:12:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:13:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:13:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:13:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:13:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:13:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:13:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:13:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:13:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:13:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:13:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:14:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:14:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:14:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:14:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:14:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:14:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:14:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:14:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:14:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:15:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:15:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:15:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:15:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:15:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:15:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:15:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:15:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:15:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:16:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:16:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:16:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:16:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:16:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:16:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:16:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:16:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:16:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:17:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:17:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:17:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:17:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:17:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:17:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:17:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:17:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:17:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:17:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:18:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:18:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:18:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:18:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:18:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:18:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:18:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:18:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:18:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:19:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:19:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:19:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:19:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:19:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:19:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:19:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:19:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:19:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:20:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:20:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:20:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:20:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:20:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:20:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:20:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:20:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:20:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:20:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:21:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:21:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:21:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:21:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:21:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:21:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:21:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:21:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:21:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:22:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:22:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:22:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:22:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:22:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:22:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:22:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:22:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:22:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:23:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:23:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:23:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:23:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:23:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:23:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:23:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:23:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:23:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:23:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:24:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:24:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:24:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:24:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:24:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:24:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:24:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:24:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:24:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:25:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:25:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:25:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:25:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:25:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:25:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:25:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:25:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:25:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:26:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:26:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:26:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:26:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:26:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:26:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:26:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:26:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:26:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:26:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:27:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:27:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:27:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:27:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:27:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:27:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:27:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:27:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:27:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:28:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:28:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:28:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:28:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:28:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:28:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:28:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:28:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:28:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:29:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:29:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:29:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:29:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:29:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:29:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:29:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:29:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:29:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:29:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:30:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:30:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:30:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:30:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:30:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:30:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:30:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:30:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:30:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:31:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:31:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:31:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:31:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:31:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:31:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:31:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:31:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:31:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:32:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:32:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:32:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:32:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:32:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:32:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:32:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:32:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:32:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:33:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:33:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:33:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:33:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:33:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:33:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:33:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:33:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:33:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:33:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:34:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:34:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:34:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:34:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:34:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:34:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:34:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:34:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:34:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:35:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:35:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:35:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:35:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:35:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:35:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:35:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:35:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:35:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:36:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:36:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:36:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:36:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:36:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:36:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:36:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:36:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:36:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:36:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:37:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:37:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:37:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:37:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:37:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:37:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:37:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:37:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:37:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:38:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:38:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:38:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:38:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:38:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:38:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:38:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:38:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:38:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:39:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:39:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:39:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:39:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:39:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:39:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:39:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:39:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:39:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:39:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:40:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:40:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:40:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:40:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:40:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:40:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:40:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:40:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:40:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:41:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:41:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:41:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:41:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:41:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:41:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:41:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:41:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:41:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:42:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  8 17:42:01 rdma06 dnf[649613]: Updating Subscription Management repositories.
Apr  8 17:42:01 rdma06 dnf[649613]: Unable to read consumer identity
Apr  8 17:42:01 rdma06 dnf[649613]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  8 17:42:01 rdma06 dnf[649613]: Metadata cache refreshed recently.
Apr  8 17:42:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  8 17:42:01 rdma06 systemd[1]: Finished dnf makecache.
Apr  8 17:42:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:42:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:42:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:42:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:42:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:42:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:42:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:42:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:42:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:42:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:43:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:43:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:43:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:43:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:43:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:43:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:43:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:43:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:43:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:44:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:44:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:44:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:44:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:44:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:44:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:44:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:44:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:44:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:45:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:45:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:45:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:45:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:45:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:45:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:45:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:45:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:45:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:45:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:46:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:46:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:46:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:46:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:46:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:46:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:46:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:46:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:46:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:47:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:47:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:47:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:47:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:47:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:47:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:47:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:47:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:47:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:48:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:48:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:48:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:48:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:48:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:48:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:48:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:48:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:48:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:49:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:49:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:49:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:49:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:49:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:49:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:49:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:49:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:49:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:49:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:50:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:50:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:50:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:50:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:50:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:50:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:50:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:50:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:50:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:51:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:51:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:51:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:51:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:51:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:51:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:51:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:51:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:51:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:52:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:52:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:52:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:52:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:52:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:52:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:52:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:52:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:52:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:52:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:53:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:53:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:53:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:53:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:53:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:53:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:53:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:53:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:53:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:54:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:54:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:54:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:54:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:54:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:54:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:54:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:54:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:54:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:55:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:55:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:55:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:55:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:55:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:55:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:55:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:55:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:55:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:55:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:56:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:56:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:56:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:56:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:56:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:56:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:56:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:56:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:56:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:57:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:57:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:57:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:57:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:57:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:57:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:57:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:57:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:57:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:58:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:58:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:58:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:58:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:58:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:58:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:58:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:58:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:58:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:58:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:59:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:59:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:59:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:59:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:59:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:59:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:59:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 17:59:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 17:59:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 17:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 17:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 17:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:00:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:00:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:00:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:00:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:00:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:00:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:00:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:00:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:00:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:01:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:01:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:01:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:01:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:01:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:01:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:01:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:01:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:01:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:01:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:02:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:02:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:02:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:02:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:02:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:02:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:02:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:02:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:02:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:03:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:03:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:03:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:03:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:03:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:03:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:03:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:03:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:03:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:04:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:04:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:04:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:04:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:04:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:04:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:04:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:04:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:04:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:04:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:05:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:05:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:05:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:05:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:05:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:05:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:05:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:05:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:05:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:06:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:06:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:06:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:06:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:06:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:06:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:06:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:06:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:06:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:07:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:07:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:07:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:07:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:07:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:07:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:07:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:07:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:07:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:08:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:08:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:08:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:08:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:08:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:08:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:08:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:08:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:08:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:08:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:09:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:09:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:09:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:09:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:09:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:09:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:09:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:09:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:09:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:10:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:10:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:10:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:10:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:10:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:10:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:10:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:10:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:10:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:11:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:11:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:11:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:11:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:11:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:11:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:11:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:11:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:11:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:11:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:12:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:12:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:12:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:12:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:12:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:12:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:12:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:12:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:12:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:13:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:13:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:13:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:13:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:13:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:13:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:13:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:13:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:13:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:14:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:14:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:14:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:14:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:14:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:14:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:14:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:14:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:14:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:14:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:15:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:15:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:15:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:15:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:15:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:15:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:15:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:15:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:15:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:16:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:16:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:16:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:16:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:16:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:16:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:16:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:16:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:16:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:17:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:17:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:17:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:17:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:17:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:17:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:17:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:17:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:17:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:17:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:18:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:18:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:18:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:18:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:18:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:18:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:18:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:18:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:18:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:19:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:19:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:19:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:19:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:19:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:19:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:19:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:19:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:19:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:20:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:20:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:20:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:20:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:20:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:20:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:20:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:20:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:20:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:20:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:21:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:21:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:21:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:21:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:21:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:21:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:21:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:21:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:21:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:22:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:22:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:22:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:22:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:22:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:22:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:22:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:22:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:22:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:23:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:23:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:23:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:23:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:23:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:23:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:23:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:23:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:23:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:24:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:24:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:24:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:24:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:24:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:24:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:24:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:24:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:24:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:24:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:25:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:25:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:25:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:25:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:25:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:25:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:25:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:25:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:25:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:26:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:26:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:26:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:26:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:26:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:26:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:26:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:26:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:26:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:27:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:27:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:27:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:27:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:27:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:27:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:27:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:27:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:27:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:27:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:28:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:28:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:28:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:28:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:28:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:28:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:28:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:28:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:28:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:29:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:29:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:29:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:29:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:29:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:29:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:29:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:29:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:29:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:30:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:30:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:30:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:30:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:30:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:30:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:30:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:30:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:30:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:30:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:31:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:31:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:31:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:31:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:31:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:31:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:31:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:31:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:31:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:32:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:32:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:32:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:32:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:32:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:32:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:32:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:32:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:32:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:33:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:33:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:33:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:33:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:33:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:33:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:33:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:33:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:33:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:33:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:34:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:34:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:34:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:34:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:34:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:34:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:34:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:34:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:34:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:35:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:35:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:35:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:35:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:35:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:35:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:35:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:35:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:35:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:36:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:36:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:36:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:36:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:36:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:36:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:36:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:36:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:36:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:36:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:37:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:37:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:37:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:37:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:37:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:37:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:37:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:37:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:37:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:38:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:38:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:38:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:38:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:38:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:38:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:38:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:38:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:38:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:39:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:39:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:39:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:39:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:39:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:39:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:39:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:39:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:39:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:40:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:40:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:40:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:40:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:40:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:40:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:40:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:40:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:40:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:40:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:41:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:41:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:41:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:41:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:41:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:41:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:41:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:41:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:41:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:42:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:42:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:42:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:42:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:42:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:42:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:42:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:42:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:42:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:43:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:43:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:43:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:43:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:43:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:43:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:43:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:43:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:43:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:43:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:44:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:44:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:44:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:44:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:44:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:44:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:44:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:44:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:44:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:45:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:45:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:45:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:45:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:45:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:45:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:45:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:45:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:45:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:46:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:46:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:46:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:46:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:46:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:46:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:46:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:46:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:46:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:46:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:47:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:47:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:47:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:47:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:47:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:47:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:47:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:47:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:47:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:48:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:48:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:48:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:48:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:48:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:48:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:48:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:48:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:48:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:49:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:49:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:49:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:49:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:49:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:49:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:49:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:49:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:49:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:49:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:50:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:50:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:50:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:50:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:50:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:50:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:50:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:50:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:50:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:51:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:51:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:51:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:51:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:51:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:51:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:51:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:51:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:51:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:52:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:52:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:52:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:52:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:52:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:52:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:52:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:52:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:52:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:52:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:53:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:53:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:53:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:53:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:53:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:53:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:53:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:53:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:53:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:54:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:54:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:54:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:54:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:54:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:54:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:54:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:54:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:54:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:55:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:55:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:55:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:55:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:55:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:55:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:55:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:55:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:55:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:56:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:56:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:56:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:56:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:56:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:56:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:56:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:56:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:56:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:56:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:57:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:57:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:57:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:57:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:57:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:57:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:57:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:57:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:57:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:58:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:58:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:58:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:58:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:58:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:58:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:58:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:58:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:58:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:59:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:59:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:59:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:59:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:59:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:59:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:59:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:59:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 18:59:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 18:59:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 18:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 18:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 18:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:00:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:00:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:00:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:00:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:00:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:00:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:00:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:00:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:00:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:01:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:01:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:01:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:01:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:01:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:01:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:01:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:01:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:01:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:02:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:02:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:02:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:02:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:02:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:02:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:02:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:02:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:02:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:02:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:03:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:03:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:03:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:03:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:03:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:03:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:03:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:03:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:03:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:04:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:04:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:04:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:04:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:04:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:04:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:04:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:04:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:04:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:05:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:05:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:05:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:05:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:05:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:05:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:05:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:05:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:05:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:05:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:06:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:06:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:06:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:06:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:06:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:06:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:06:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:06:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:06:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:07:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:07:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:07:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:07:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:07:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:07:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:07:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:07:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:07:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:08:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:08:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:08:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:08:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:08:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:08:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:08:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:08:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:08:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:08:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:09:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:09:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:09:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:09:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:09:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:09:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:09:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:09:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:09:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:10:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:10:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:10:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:10:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:10:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:10:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:10:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:10:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:10:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:11:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:11:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:11:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:11:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:11:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:11:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:11:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:11:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:11:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:11:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:12:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:12:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:12:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:12:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:12:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:12:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:12:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:12:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:12:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:13:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:13:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:13:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:13:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:13:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:13:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:13:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:13:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:13:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:14:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  8 19:14:01 rdma06 dnf[653343]: Updating Subscription Management repositories.
Apr  8 19:14:01 rdma06 dnf[653343]: Unable to read consumer identity
Apr  8 19:14:01 rdma06 dnf[653343]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  8 19:14:01 rdma06 dnf[653343]: Metadata cache refreshed recently.
Apr  8 19:14:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  8 19:14:01 rdma06 systemd[1]: Finished dnf makecache.
Apr  8 19:14:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:14:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:14:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:14:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:14:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:14:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:14:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:14:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:14:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:15:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:15:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:15:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:15:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:15:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:15:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:15:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:15:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:15:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:15:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:16:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:16:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:16:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:16:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:16:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:16:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:16:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:16:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:16:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:17:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:17:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:17:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:17:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:17:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:17:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:17:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:17:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:17:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:18:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:18:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:18:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:18:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:18:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:18:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:18:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:18:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:18:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:18:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:19:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:19:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:19:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:19:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:19:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:19:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:19:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:19:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:19:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:20:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:20:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:20:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:20:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:20:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:20:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:20:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:20:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:20:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:21:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:21:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:21:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:21:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:21:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:21:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:21:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:21:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:21:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:21:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:22:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:22:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:22:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:22:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:22:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:22:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:22:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:22:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:22:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:23:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:23:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:23:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:23:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:23:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:23:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:23:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:23:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:23:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:24:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:24:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:24:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:24:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:24:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:24:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:24:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:24:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:24:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:24:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:25:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:25:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:25:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:25:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:25:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:25:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:25:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:25:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:25:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:26:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:26:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:26:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:26:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:26:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:26:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:26:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:26:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:26:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:27:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:27:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:27:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:27:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:27:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:27:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:27:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:27:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:27:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:27:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:28:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:28:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:28:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:28:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:28:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:28:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:28:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:28:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:28:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:29:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:29:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:29:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:29:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:29:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:29:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:29:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:29:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:29:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:30:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:30:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:30:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:30:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:30:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:30:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:30:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:30:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:30:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:31:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:31:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:31:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:31:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:31:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:31:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:31:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:31:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:31:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:31:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:32:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:32:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:32:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:32:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:32:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:32:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:32:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:32:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:32:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:33:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:33:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:33:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:33:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:33:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:33:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:33:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:33:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:33:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:34:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:34:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:34:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:34:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:34:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:34:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:34:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:34:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:34:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:34:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:35:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:35:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:35:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:35:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:35:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:35:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:35:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:35:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:35:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:36:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:36:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:36:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:36:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:36:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:36:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:36:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:36:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:36:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:37:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:37:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:37:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:37:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:37:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:37:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:37:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:37:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:37:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:37:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:38:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:38:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:38:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:38:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:38:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:38:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:38:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:38:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:38:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:39:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:39:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:39:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:39:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:39:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:39:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:39:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:39:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:39:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:40:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:40:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:40:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:40:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:40:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:40:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:40:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:40:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:40:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:40:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:41:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:41:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:41:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:41:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:41:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:41:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:41:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:41:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:41:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:42:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:42:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:42:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:42:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:42:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:42:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:42:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:42:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:42:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:43:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:43:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:43:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:43:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:43:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:43:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:43:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:43:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:43:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:43:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:44:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:44:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:44:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:44:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:44:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:44:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:44:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:44:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:44:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:45:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:45:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:45:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:45:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:45:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:45:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:45:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:45:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:45:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:46:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:46:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:46:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:46:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:46:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:46:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:46:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:46:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:46:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:47:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:47:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:47:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:47:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:47:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:47:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:47:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:47:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:47:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:47:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:48:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:48:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:48:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:48:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:48:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:48:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:48:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:48:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:48:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:49:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:49:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:49:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:49:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:49:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:49:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:49:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:49:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:49:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:50:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:50:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:50:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:50:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:50:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:50:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:50:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:50:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:50:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:50:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:51:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:51:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:51:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:51:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:51:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:51:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:51:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:51:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:51:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:52:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:52:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:52:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:52:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:52:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:52:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:52:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:52:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:52:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:53:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:53:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:53:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:53:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:53:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:53:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:53:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:53:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:53:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:53:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:54:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:54:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:54:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:54:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:54:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:54:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:54:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:54:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:54:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:55:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:55:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:55:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:55:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:55:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:55:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:55:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:55:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:55:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:56:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:56:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:56:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:56:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:56:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:56:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:56:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:56:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:56:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:56:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:57:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:57:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:57:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:57:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:57:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:57:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:57:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:57:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:57:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:58:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:58:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:58:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:58:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:58:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:58:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:58:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:58:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:58:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:59:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:59:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:59:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:59:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:59:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:59:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:59:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 19:59:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:59:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 19:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 19:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 19:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 19:59:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:00:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:00:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:00:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:00:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:00:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:00:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:00:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:00:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:00:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:01:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:01:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:01:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:01:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:01:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:01:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:01:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:01:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:01:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:02:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:02:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:02:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:02:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:02:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:02:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:02:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:02:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:02:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:03:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:03:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:03:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:03:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:03:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:03:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:03:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:03:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:03:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:03:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:04:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:04:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:04:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:04:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:04:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:04:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:04:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:04:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:04:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:05:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:05:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:05:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:05:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:05:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:05:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:05:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:05:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:05:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:06:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:06:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:06:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:06:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:06:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:06:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:06:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:06:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:06:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:06:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:07:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:07:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:07:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:07:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:07:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:07:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:07:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:07:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:07:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:08:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:08:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:08:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:08:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:08:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:08:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:08:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:08:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:08:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:09:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:09:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:09:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:09:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:09:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:09:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:09:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:09:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:09:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:09:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:10:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:10:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:10:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:10:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:10:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:10:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:10:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:10:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:10:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:11:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:11:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:11:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:11:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:11:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:11:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:11:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:11:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:11:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:12:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:12:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:12:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:12:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:12:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:12:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:12:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:12:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:12:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:12:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:13:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:13:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:13:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:13:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:13:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:13:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:13:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:13:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:13:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:14:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:14:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:14:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:14:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:14:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:14:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:14:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:14:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:14:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:15:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:15:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:15:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:15:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:15:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:15:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:15:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:15:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:15:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:15:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:16:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:16:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:16:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:16:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:16:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:16:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:16:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:16:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:16:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:17:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:17:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:17:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:17:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:17:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:17:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:17:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:17:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:17:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:18:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:18:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:18:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:18:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:18:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:18:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:18:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:18:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:18:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:18:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:19:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:19:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:19:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:19:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:19:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:19:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:19:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:19:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:19:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:20:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:20:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:20:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:20:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:20:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:20:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:20:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:20:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:20:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:21:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:21:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:21:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:21:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:21:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:21:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:21:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:21:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:21:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:22:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:22:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:22:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:22:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:22:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:22:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:22:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:22:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:22:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:22:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:23:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:23:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:23:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:23:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:23:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:23:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:23:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:23:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:23:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:24:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:24:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:24:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:24:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:24:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:24:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:24:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:24:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:24:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:25:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:25:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:25:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:25:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:25:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:25:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:25:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:25:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:25:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:25:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:26:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:26:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:26:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:26:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:26:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:26:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:26:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:26:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:26:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:27:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:27:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:27:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:27:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:27:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:27:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:27:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:27:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:27:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:28:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:28:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:28:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:28:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:28:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:28:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:28:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:28:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:28:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:28:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:29:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:29:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:29:15 rdma06 systemd[1]: Starting dnf makecache...
Apr  8 20:29:15 rdma06 dnf[656384]: Updating Subscription Management repositories.
Apr  8 20:29:15 rdma06 dnf[656384]: Unable to read consumer identity
Apr  8 20:29:15 rdma06 dnf[656384]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  8 20:29:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:29:17 rdma06 dnf[656384]: Copr repo for fmf owned by psss                 2.4 kB/s | 3.3 kB     00:01
Apr  8 20:29:17 rdma06 dnf[656384]: beaker-AppStream                                 78 kB/s | 3.2 kB     00:00
Apr  8 20:29:17 rdma06 dnf[656384]: beaker-AppStream-debuginfo                       36 kB/s | 1.5 kB     00:00
Apr  8 20:29:17 rdma06 dnf[656384]: beaker-BaseOS                                   134 kB/s | 2.7 kB     00:00
Apr  8 20:29:17 rdma06 dnf[656384]: beaker-BaseOS-debuginfo                          38 kB/s | 1.5 kB     00:00
Apr  8 20:29:17 rdma06 dnf[656384]: beaker-CRB                                       64 kB/s | 2.7 kB     00:00
Apr  8 20:29:18 rdma06 dnf[656384]: beaker-CRB-debuginfo                            7.3 kB/s | 1.5 kB     00:00
Apr  8 20:29:18 rdma06 dnf[656384]: beaker-HighAvailability                          70 kB/s | 2.8 kB     00:00
Apr  8 20:29:18 rdma06 dnf[656384]: beaker-HighAvailability-debuginfo                38 kB/s | 1.5 kB     00:00
Apr  8 20:29:18 rdma06 dnf[656384]: beaker-NFV                                       65 kB/s | 2.7 kB     00:00
Apr  8 20:29:18 rdma06 dnf[656384]: beaker-NFV-debuginfo                             32 kB/s | 1.5 kB     00:00
Apr  8 20:29:18 rdma06 dnf[656384]: beaker-RT                                        49 kB/s | 2.7 kB     00:00
Apr  8 20:29:18 rdma06 dnf[656384]: beaker-RT-debuginfo                              54 kB/s | 1.5 kB     00:00
Apr  8 20:29:18 rdma06 dnf[656384]: beaker-ResilientStorage                          70 kB/s | 2.8 kB     00:00
Apr  8 20:29:18 rdma06 dnf[656384]: beaker-ResilientStorage-debuginfo                57 kB/s | 1.5 kB     00:00
Apr  8 20:29:18 rdma06 dnf[656384]: beaker-SAP                                       61 kB/s | 2.7 kB     00:00
Apr  8 20:29:18 rdma06 dnf[656384]: beaker-SAP-debuginfo                             76 kB/s | 1.5 kB     00:00
Apr  8 20:29:18 rdma06 dnf[656384]: beaker-SAPHANA                                   47 kB/s | 2.7 kB     00:00
Apr  8 20:29:18 rdma06 dnf[656384]: beaker-SAPHANA-debuginfo                         50 kB/s | 1.5 kB     00:00
Apr  8 20:29:19 rdma06 dnf[656384]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr  8 20:29:20 rdma06 dnf[656384]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr  8 20:29:20 rdma06 dnf[656384]: Metadata cache created.
Apr  8 20:29:20 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  8 20:29:20 rdma06 systemd[1]: Finished dnf makecache.
Apr  8 20:29:20 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.405s CPU time.
Apr  8 20:29:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:29:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:29:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:29:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:29:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:29:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:30:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:30:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:30:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:30:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:30:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:30:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:30:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:30:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:30:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:31:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:31:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:31:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:31:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:31:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:31:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:31:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:31:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:31:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:31:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:32:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:32:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:32:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:32:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:32:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:32:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:32:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:32:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:32:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:33:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:33:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:33:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:33:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:33:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:33:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:33:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:33:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:33:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:34:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:34:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:34:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:34:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:34:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:34:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:34:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:34:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:34:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:34:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:35:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:35:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:35:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:35:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:35:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:35:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:35:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:35:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:35:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:36:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:36:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:36:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:36:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:36:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:36:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:36:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:36:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:36:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:37:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:37:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:37:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:37:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:37:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:37:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:37:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:37:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:37:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:38:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:38:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:38:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:38:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:38:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:38:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:38:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:38:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:38:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:38:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:39:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:39:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:39:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:39:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:39:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:39:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:39:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:39:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:39:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:40:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:40:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:40:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:40:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:40:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:40:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:40:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:40:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:40:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:41:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:41:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:41:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:41:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:41:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:41:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:41:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:41:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:41:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:41:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:42:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:42:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:42:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:42:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:42:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:42:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:42:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:42:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:42:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:43:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:43:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:43:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:43:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:43:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:43:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:43:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:43:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:43:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:44:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:44:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:44:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:44:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:44:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:44:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:44:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:44:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:44:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:44:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:45:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:45:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:45:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:45:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:45:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:45:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:45:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:45:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:45:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:46:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:46:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:46:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:46:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:46:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:46:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:46:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:46:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:46:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:47:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:47:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:47:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:47:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:47:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:47:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:47:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:47:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:47:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:47:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:48:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:48:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:48:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:48:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:48:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:48:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:48:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:48:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:48:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:49:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:49:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:49:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:49:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:49:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:49:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:49:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:49:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:49:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:50:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:50:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:50:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:50:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:50:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:50:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:50:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:50:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:50:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:50:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:51:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:51:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:51:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:51:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:51:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:51:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:51:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:51:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:51:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:52:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:52:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:52:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:52:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:52:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:52:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:52:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:52:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:52:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:53:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:53:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:53:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:53:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:53:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:53:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:53:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:53:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:53:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:54:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:54:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:54:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:54:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:54:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:54:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:54:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:54:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:54:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:54:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:55:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:55:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:55:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:55:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:55:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:55:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:55:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:55:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:55:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:56:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:56:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:56:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:56:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:56:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:56:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:56:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:56:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:56:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:57:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:57:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:57:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:57:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:57:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:57:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:57:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:57:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:57:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:57:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:58:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:58:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:58:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:58:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:58:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:58:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:58:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:58:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:58:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:59:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:59:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:59:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:59:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:59:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:59:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:59:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 20:59:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 20:59:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 20:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 20:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 20:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:00:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:00:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:00:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:00:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:00:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:00:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:00:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:00:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:00:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:00:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:01:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:01:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:01:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:01:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:01:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:01:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:01:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:01:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:01:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:02:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:02:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:02:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:02:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:02:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:02:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:02:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:02:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:02:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:03:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:03:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:03:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:03:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:03:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:03:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:03:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:03:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:03:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:03:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:04:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:04:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:04:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:04:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:04:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:04:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:04:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:04:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:04:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:05:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:05:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:05:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:05:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:05:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:05:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:05:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:05:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:05:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:06:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:06:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:06:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:06:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:06:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:06:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:06:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:06:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:06:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:06:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:07:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:07:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:07:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:07:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:07:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:07:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:07:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:07:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:07:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:08:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:08:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:08:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:08:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:08:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:08:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:08:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:08:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:08:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:09:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:09:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:09:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:09:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:09:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:09:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:09:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:09:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:09:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:10:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:10:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:10:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:10:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:10:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:10:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:10:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:10:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:10:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:10:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:11:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:11:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:11:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:11:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:11:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:11:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:11:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:11:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:11:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:12:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:12:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:12:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:12:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:12:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:12:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:12:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:12:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:12:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:13:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:13:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:13:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:13:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:13:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:13:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:13:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:13:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:13:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:13:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:14:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:14:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:14:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:14:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:14:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:14:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:14:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:14:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:14:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:15:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:15:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:15:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:15:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:15:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:15:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:15:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:15:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:15:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:16:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:16:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:16:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:16:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:16:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:16:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:16:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:16:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:16:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:16:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:17:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:17:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:17:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:17:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:17:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:17:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:17:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:17:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:17:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:18:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:18:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:18:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:18:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:18:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:18:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:18:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:18:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:18:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:19:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:19:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:19:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:19:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:19:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:19:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:19:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:19:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:19:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:19:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:20:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:20:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:20:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:20:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:20:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:20:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:20:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:20:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:20:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:21:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:21:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:21:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:21:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:21:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:21:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:21:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:21:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:21:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:22:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:22:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:22:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:22:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:22:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:22:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:22:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:22:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:22:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:22:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:23:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:23:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:23:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:23:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:23:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:23:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:23:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:23:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:23:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:24:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:24:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:24:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:24:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:24:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:24:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:24:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:24:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:24:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:25:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:25:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:25:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:25:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:25:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:25:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:25:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:25:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:25:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:25:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:26:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:26:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:26:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:26:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:26:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:26:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:26:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:26:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:26:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:27:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:27:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:27:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:27:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:27:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:27:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:27:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:27:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:27:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:28:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:28:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:28:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:28:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:28:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:28:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:28:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:28:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:28:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:29:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:29:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:29:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:29:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:29:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:29:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:29:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:29:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:29:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:29:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:30:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:30:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:30:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:30:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:30:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:30:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:30:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:30:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:30:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:31:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:31:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:31:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:31:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:31:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:31:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:31:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:31:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:31:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:32:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:32:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:32:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:32:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:32:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:32:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:32:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:32:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:32:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:32:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:33:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:33:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:33:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:33:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:33:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:33:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:33:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:33:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:33:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:34:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:34:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:34:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:34:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:34:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:34:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:34:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:34:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:34:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:35:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:35:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:35:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:35:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:35:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:35:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:35:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:35:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:35:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:35:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:36:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:36:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:36:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:36:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:36:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:36:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:36:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:36:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:36:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:37:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:37:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:37:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:37:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:37:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:37:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:37:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:37:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:37:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:38:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:38:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:38:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:38:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:38:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:38:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:38:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:38:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:38:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:38:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:39:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:39:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:39:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:39:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:39:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:39:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:39:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:39:46 rdma06 systemd[1]: Starting dnf makecache...
Apr  8 21:39:47 rdma06 dnf[659261]: Updating Subscription Management repositories.
Apr  8 21:39:47 rdma06 dnf[659261]: Unable to read consumer identity
Apr  8 21:39:47 rdma06 dnf[659261]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  8 21:39:47 rdma06 dnf[659261]: Metadata cache refreshed recently.
Apr  8 21:39:47 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  8 21:39:47 rdma06 systemd[1]: Finished dnf makecache.
Apr  8 21:39:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:39:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:40:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:40:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:40:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:40:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:40:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:40:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:40:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:40:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:40:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:41:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:41:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:41:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:41:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:41:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:41:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:41:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:41:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:41:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:41:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:42:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:42:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:42:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:42:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:42:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:42:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:42:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:42:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:42:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:43:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:43:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:43:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:43:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:43:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:43:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:43:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:43:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:43:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:44:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:44:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:44:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:44:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:44:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:44:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:44:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:44:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:44:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:45:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:45:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:45:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:45:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:45:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:45:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:45:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:45:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:45:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:45:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:46:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:46:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:46:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:46:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:46:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:46:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:46:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:46:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:46:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:47:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:47:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:47:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:47:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:47:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:47:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:47:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:47:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:47:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:48:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:48:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:48:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:48:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:48:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:48:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:48:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:48:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:48:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:48:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:49:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:49:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:49:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:49:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:49:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:49:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:49:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:49:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:49:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:50:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:50:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:50:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:50:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:50:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:50:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:50:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:50:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:50:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:51:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:51:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:51:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:51:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:51:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:51:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:51:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:51:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:51:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:51:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:52:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:52:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:52:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:52:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:52:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:52:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:52:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:52:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:52:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:53:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:53:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:53:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:53:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:53:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:53:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:53:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:53:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:53:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:54:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:54:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:54:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:54:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:54:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:54:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:54:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:54:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:54:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:54:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:55:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:55:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:55:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:55:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:55:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:55:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:55:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:55:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:55:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:56:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:56:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:56:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:56:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:56:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:56:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:56:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:56:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:56:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:57:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:57:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:57:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:57:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:57:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:57:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:57:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:57:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:57:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:57:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:58:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:58:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:58:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:58:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:58:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:58:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:58:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:58:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:58:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:59:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:59:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:59:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:59:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:59:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:59:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:59:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 21:59:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 21:59:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 21:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 21:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 21:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:00:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:00:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:00:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:00:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:00:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:00:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:00:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:00:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:00:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:01:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:01:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:01:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:01:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:01:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:01:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:01:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:01:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:01:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:01:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:02:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:02:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:02:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:02:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:02:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:02:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:02:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:02:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:02:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:03:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:03:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:03:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:03:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:03:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:03:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:03:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:03:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:03:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:04:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:04:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:04:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:04:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:04:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:04:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:04:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:04:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:04:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:04:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:05:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:05:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:05:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:05:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:05:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:05:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:05:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:05:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:05:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:06:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:06:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:06:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:06:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:06:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:06:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:06:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:06:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:06:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:07:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:07:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:07:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:07:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:07:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:07:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:07:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:07:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:07:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:07:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:08:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:08:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:08:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:08:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:08:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:08:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:08:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:08:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:08:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:09:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:09:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:09:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:09:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:09:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:09:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:09:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:09:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:09:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:10:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:10:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:10:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:10:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:10:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:10:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:10:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:10:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:10:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:10:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:11:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:11:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:11:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:11:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:11:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:11:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:11:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:11:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:11:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:12:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:12:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:12:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:12:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:12:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:12:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:12:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:12:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:12:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:13:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:13:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:13:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:13:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:13:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:13:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:13:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:13:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:13:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:13:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:14:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:14:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:14:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:14:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:14:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:14:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:14:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:14:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:14:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:14:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:15:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:15:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:15:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:15:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:15:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:15:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:15:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:15:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:15:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:16:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:16:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:16:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:16:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:16:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:16:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:16:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:16:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:16:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:17:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:17:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:17:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:17:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:17:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:17:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:17:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:17:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:17:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:17:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:18:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:18:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:18:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:18:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:18:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:18:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:18:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:18:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:18:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:19:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:19:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:19:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:19:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:19:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:19:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:19:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:19:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:19:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:20:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:20:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:20:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:20:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:20:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:20:32 rdma06 NetworkManager[1025]: <info>  [1681006832.9367] dhcp4 (eno1): state changed new lease, address=10.73.131.45
Apr  8 22:20:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:20:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:20:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:20:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:20:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:21:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:21:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:21:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:21:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:21:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:21:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:21:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:21:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:21:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:22:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:22:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:22:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:22:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:22:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:22:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:22:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:22:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:22:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:23:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:23:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:23:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:23:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:23:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:23:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:23:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:23:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:23:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:23:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:23:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:24:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:24:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:24:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:24:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:24:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:24:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:24:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:24:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:24:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:24:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:25:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:25:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:25:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:25:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:25:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:25:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:25:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:25:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:25:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:26:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:26:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:26:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:26:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:26:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:26:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:26:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:26:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:26:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:26:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:27:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:27:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:27:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:27:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:27:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:27:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:27:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:27:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:27:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:27:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:27:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:28:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:28:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:28:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:28:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:28:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:28:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:28:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:28:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:28:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:29:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:29:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:29:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:29:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:29:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:29:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:29:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:29:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:29:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:29:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:29:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:30:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:30:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:30:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:30:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:30:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:30:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:30:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:30:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:30:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:30:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:30:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:31:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:31:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:31:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:31:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:31:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:31:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:31:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:31:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:31:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:31:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:31:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:32:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:32:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:32:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:32:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:32:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:32:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:32:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:32:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:32:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:32:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:33:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:33:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:33:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:33:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:33:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:33:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:33:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:33:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:33:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:34:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:34:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:34:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:34:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:34:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:34:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:34:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:34:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:34:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:35:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:35:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:35:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:35:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:35:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:35:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:35:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:35:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:35:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:35:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:35:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:36:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:36:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:36:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:36:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:36:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:36:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:36:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:36:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:36:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:36:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:37:00 rdma06 systemd[5685]: Starting Cleanup of User's Temporary Files and Directories...
Apr  8 22:37:00 rdma06 systemd[5685]: Finished Cleanup of User's Temporary Files and Directories.
Apr  8 22:37:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:37:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:37:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:37:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:37:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:37:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:37:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:37:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:37:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:38:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:38:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:38:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:38:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:38:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:38:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:38:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:38:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:38:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:39:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:39:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:39:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:39:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:39:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:39:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:39:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:39:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:39:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:39:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:39:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:40:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:40:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:40:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:40:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:40:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:40:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:40:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:40:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:40:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:40:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:41:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:41:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:41:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:41:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:41:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:41:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:41:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:41:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:41:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:41:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:41:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:42:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:42:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:42:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:42:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:42:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:42:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:42:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:42:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:42:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:42:46 rdma06 systemd[1]: Starting Cleanup of Temporary Directories...
Apr  8 22:42:46 rdma06 systemd[1]: systemd-tmpfiles-clean.service: Deactivated successfully.
Apr  8 22:42:46 rdma06 systemd[1]: Finished Cleanup of Temporary Directories.
Apr  8 22:42:47 rdma06 systemd[1]: run-credentials-systemd\x2dtmpfiles\x2dclean.service.mount: Deactivated successfully.
Apr  8 22:42:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:42:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:43:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:43:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:43:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:43:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:43:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:43:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:43:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:43:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:43:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:43:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:43:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:44:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:44:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:44:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:44:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:44:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:44:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:44:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:44:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:44:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:45:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:45:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:45:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:45:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:45:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:45:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:45:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:45:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:45:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:45:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:46:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:46:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:46:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:46:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:46:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:46:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:46:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:46:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:46:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:46:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:47:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:47:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:47:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:47:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:47:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:47:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:47:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:47:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:47:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:47:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:48:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:48:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:48:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:48:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:48:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:48:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:48:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:48:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:48:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:48:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:49:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:49:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:49:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:49:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:49:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:49:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:49:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:49:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:49:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:49:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:50:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:50:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:50:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:50:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:50:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:50:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:50:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:50:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:50:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:51:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:51:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:51:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:51:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:51:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:51:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:51:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:51:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:51:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:51:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:51:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:52:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:52:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:52:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:52:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:52:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:52:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:52:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:52:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:52:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:52:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:52:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:52:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:53:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:53:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:53:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:53:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:53:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:53:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:53:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:53:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:53:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:53:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:53:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:54:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:54:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:54:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:54:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:54:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:54:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:54:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:54:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:54:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:54:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:54:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:54:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:55:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:55:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:55:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:55:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:55:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:55:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:55:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:55:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:55:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:55:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:55:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:55:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:56:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:56:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:56:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:56:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:56:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:56:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:56:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:56:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:56:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:56:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:56:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:56:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:57:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:57:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:57:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:57:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:57:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:57:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:57:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:57:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:57:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:57:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:57:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:57:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:58:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:58:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:58:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:58:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:58:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:58:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:58:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:58:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:58:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:58:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:58:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:58:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:58:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:58:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:59:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:59:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:59:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:59:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:59:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:59:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:59:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:59:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:59:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:59:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 22:59:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 22:59:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 22:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 22:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 22:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:00:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:00:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:00:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:00:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:00:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:00:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:00:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:00:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:00:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:00:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:01:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:01:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:01:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:01:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:01:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:01:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:01:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:01:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:01:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:01:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:01:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:01:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:01:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:01:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:01:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:01:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:02:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:02:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:02:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:02:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:02:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:02:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:02:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:02:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:02:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:02:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:02:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:03:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:03:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:03:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:03:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:03:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:03:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:03:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:03:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:03:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:03:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:03:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:04:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:04:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:04:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:04:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:04:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:04:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:04:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:04:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:04:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:04:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:04:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:04:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:04:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:05:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:05:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:05:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:05:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:05:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:05:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:05:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:05:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:05:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:05:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:05:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:05:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:05:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:05:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:06:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:06:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:06:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:06:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:06:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:06:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:06:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:06:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:06:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:06:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:06:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:06:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:06:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:06:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:06:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:07:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:07:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:07:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:07:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:07:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:07:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:07:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:07:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:07:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:07:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:07:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:07:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:07:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:08:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:08:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:08:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:08:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:08:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:08:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:08:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:08:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:08:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:08:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:08:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:08:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:08:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:09:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:09:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:09:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:09:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:09:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:09:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:09:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:09:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:09:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:09:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:09:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:09:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:09:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:09:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:09:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:10:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:10:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:10:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:10:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:10:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:10:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:10:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:10:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:10:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:10:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:10:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:10:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:10:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:10:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:10:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:11:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:11:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:11:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:11:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:11:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:11:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:11:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:11:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:11:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:11:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:11:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:11:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:11:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:11:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:12:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:12:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:12:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:12:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:12:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:12:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:12:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:12:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:12:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:12:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:12:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:12:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:12:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:12:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:12:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:13:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:13:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:13:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:13:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:13:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:13:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:13:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:13:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:13:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:13:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:13:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:13:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:13:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:13:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:14:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:14:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:14:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:14:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:14:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:14:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:14:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:14:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:14:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:14:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:14:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:14:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:14:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:14:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:15:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:15:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:15:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:15:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:15:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:15:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:15:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:15:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:15:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:15:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:15:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:15:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:15:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:15:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:15:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:16:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:16:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:16:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:16:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:16:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:16:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:16:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:16:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:16:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:16:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:16:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:16:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:16:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:16:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:16:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:17:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:17:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:17:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:17:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:17:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:17:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:17:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:17:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:17:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:17:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:17:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:17:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:17:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:17:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:17:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:17:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:18:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:18:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:18:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:18:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:18:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:18:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:18:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:18:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:18:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:18:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:18:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:18:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:18:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:18:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:18:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:19:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:19:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:19:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:19:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:19:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:19:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:19:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:19:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:19:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:19:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:19:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:19:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:19:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:19:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:19:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:20:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:20:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:20:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:20:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:20:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:20:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:20:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:20:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:20:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:20:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:20:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:20:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:20:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:20:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:20:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:20:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:21:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:21:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:21:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:21:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:21:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:21:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:21:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:21:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:21:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:21:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:21:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:21:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:21:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:21:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:21:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:22:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:22:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:22:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:22:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:22:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:22:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:22:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:22:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:22:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:22:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:22:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:22:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:22:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:22:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:22:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:23:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:23:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:23:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:23:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:23:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:23:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:23:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:23:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:23:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:23:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:23:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:23:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:23:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:23:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:24:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:24:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:24:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:24:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:24:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:24:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:24:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:24:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:24:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:24:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:24:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:24:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:24:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:24:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:24:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:25:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:25:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:25:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:25:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:25:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:25:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:25:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:25:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:25:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:25:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:25:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:25:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:25:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:25:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:25:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:26:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:26:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:26:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:26:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:26:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:26:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:26:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:26:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:26:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:26:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:26:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:26:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:26:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:26:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:26:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:27:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:27:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:27:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:27:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:27:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:27:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:27:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:27:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:27:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:27:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:27:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:27:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:27:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:27:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:28:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:28:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:28:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:28:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:28:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:28:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:28:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:28:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:28:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:28:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:28:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:28:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:28:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:28:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:28:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:29:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:29:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:29:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:29:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:29:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:29:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:29:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:29:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:29:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:29:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:29:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:29:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:29:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:29:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:30:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:30:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:30:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:30:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:30:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:30:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:30:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:30:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:30:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:30:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:30:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:30:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:30:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:30:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:31:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:31:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:31:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:31:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:31:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:31:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:31:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:31:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:31:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:31:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:31:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:31:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:31:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:32:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:32:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:32:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:32:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:32:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:32:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:32:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:32:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:32:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:32:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:32:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:32:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:32:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:32:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:32:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:33:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:33:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:33:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:33:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:33:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:33:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:33:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:33:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:33:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:33:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:33:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:33:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:33:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:33:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:33:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:33:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:34:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:34:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:34:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:34:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:34:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:34:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:34:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:34:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:34:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:34:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:34:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:34:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:34:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:34:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:34:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:35:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:35:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:35:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:35:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:35:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:35:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:35:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:35:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:35:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:35:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:35:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:35:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:35:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:36:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:36:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:36:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:36:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:36:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:36:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:36:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:36:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:36:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:36:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:36:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:36:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:36:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:36:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:36:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:36:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:37:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  8 23:37:01 rdma06 dnf[664013]: Updating Subscription Management repositories.
Apr  8 23:37:01 rdma06 dnf[664013]: Unable to read consumer identity
Apr  8 23:37:01 rdma06 dnf[664013]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  8 23:37:03 rdma06 dnf[664013]: Copr repo for fmf owned by psss                 1.8 kB/s | 3.3 kB     00:01
Apr  8 23:37:03 rdma06 dnf[664013]: beaker-AppStream                                 57 kB/s | 3.2 kB     00:00
Apr  8 23:37:03 rdma06 dnf[664013]: beaker-AppStream-debuginfo                       39 kB/s | 1.5 kB     00:00
Apr  8 23:37:03 rdma06 dnf[664013]: beaker-BaseOS                                    74 kB/s | 2.7 kB     00:00
Apr  8 23:37:03 rdma06 dnf[664013]: beaker-BaseOS-debuginfo                          40 kB/s | 1.5 kB     00:00
Apr  8 23:37:03 rdma06 dnf[664013]: beaker-CRB                                       55 kB/s | 2.7 kB     00:00
Apr  8 23:37:03 rdma06 dnf[664013]: beaker-CRB-debuginfo                             73 kB/s | 1.5 kB     00:00
Apr  8 23:37:03 rdma06 dnf[664013]: beaker-HighAvailability                          63 kB/s | 2.8 kB     00:00
Apr  8 23:37:03 rdma06 dnf[664013]: beaker-HighAvailability-debuginfo                44 kB/s | 1.5 kB     00:00
Apr  8 23:37:04 rdma06 dnf[664013]: beaker-NFV                                       78 kB/s | 2.7 kB     00:00
Apr  8 23:37:04 rdma06 dnf[664013]: beaker-NFV-debuginfo                             64 kB/s | 1.5 kB     00:00
Apr  8 23:37:04 rdma06 dnf[664013]: beaker-RT                                        68 kB/s | 2.7 kB     00:00
Apr  8 23:37:04 rdma06 dnf[664013]: beaker-RT-debuginfo                              70 kB/s | 1.5 kB     00:00
Apr  8 23:37:04 rdma06 dnf[664013]: beaker-ResilientStorage                          82 kB/s | 2.8 kB     00:00
Apr  8 23:37:04 rdma06 dnf[664013]: beaker-ResilientStorage-debuginfo                52 kB/s | 1.5 kB     00:00
Apr  8 23:37:04 rdma06 dnf[664013]: beaker-SAP                                       72 kB/s | 2.7 kB     00:00
Apr  8 23:37:04 rdma06 dnf[664013]: beaker-SAP-debuginfo                             59 kB/s | 1.5 kB     00:00
Apr  8 23:37:04 rdma06 dnf[664013]: beaker-SAPHANA                                   61 kB/s | 2.7 kB     00:00
Apr  8 23:37:04 rdma06 dnf[664013]: beaker-SAPHANA-debuginfo                         32 kB/s | 1.5 kB     00:00
Apr  8 23:37:05 rdma06 dnf[664013]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr  8 23:37:05 rdma06 dnf[664013]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr  8 23:37:06 rdma06 dnf[664013]: Metadata cache created.
Apr  8 23:37:06 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  8 23:37:06 rdma06 systemd[1]: Finished dnf makecache.
Apr  8 23:37:06 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.400s CPU time.
Apr  8 23:37:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:37:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:37:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:37:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:37:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:37:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:37:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:37:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:37:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:37:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:37:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:37:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:37:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:37:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:37:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:38:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:38:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:38:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:38:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:38:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:38:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:38:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:38:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:38:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:38:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:38:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:38:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:38:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:38:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:38:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:39:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:39:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:39:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:39:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:39:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:39:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:39:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:39:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:39:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:39:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:39:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:39:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:39:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:39:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:39:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:40:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:40:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:40:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:40:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:40:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:40:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:40:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:40:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:40:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:40:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:40:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:40:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:40:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:40:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:41:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:41:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:41:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:41:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:41:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:41:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:41:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:41:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:41:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:41:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:41:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:41:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:41:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:42:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:42:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:42:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:42:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:42:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:42:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:42:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:42:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:42:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:42:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:42:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:42:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:42:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:42:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:43:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:43:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:43:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:43:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:43:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:43:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:43:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:43:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:43:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:43:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:43:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:43:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:43:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:43:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:44:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:44:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:44:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:44:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:44:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:44:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:44:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:44:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:44:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:44:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:44:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:44:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:44:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:44:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:44:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:45:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:45:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:45:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:45:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:45:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:45:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:45:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:45:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:45:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:45:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:45:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:45:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:45:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:45:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:45:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:46:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:46:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:46:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:46:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:46:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:46:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:46:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:46:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:46:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:46:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:46:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:46:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:46:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:46:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:46:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:47:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:47:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:47:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:47:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:47:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:47:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:47:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:47:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:47:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:47:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:47:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:47:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:47:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:47:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:48:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:48:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:48:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:48:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:48:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:48:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:48:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:48:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:48:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:48:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:48:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:48:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:48:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:48:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:48:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:49:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:49:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:49:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:49:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:49:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:49:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:49:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:49:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:49:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:49:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:49:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:49:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:49:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:49:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:49:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:50:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:50:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:50:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:50:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:50:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:50:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:50:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:50:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:50:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:50:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:50:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:50:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:50:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:50:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:50:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:51:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:51:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:51:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:51:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:51:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:51:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:51:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:51:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:51:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:51:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:51:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:51:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:51:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:52:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:52:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:52:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:52:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:52:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:52:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:52:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:52:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:52:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:52:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:52:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:52:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:52:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:52:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:53:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:53:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:53:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:53:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:53:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:53:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:53:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:53:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:53:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:53:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:53:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:53:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:53:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:54:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:54:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:54:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:54:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:54:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:54:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:54:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:54:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:54:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:54:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:54:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:54:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:55:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:55:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:55:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:55:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:55:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:55:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:55:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:55:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:55:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:55:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:55:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:55:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:55:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:55:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:56:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:56:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:56:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:56:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:56:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:56:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:56:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:56:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:56:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:56:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:56:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:56:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:57:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:57:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:57:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:57:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:57:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:57:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:57:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:57:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:57:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:57:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:57:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:57:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:58:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:58:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:58:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:58:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:58:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:58:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:58:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:58:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:58:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:58:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:58:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:59:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:59:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:59:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:59:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:59:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:59:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:59:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:59:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:59:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:59:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  8 23:59:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  8 23:59:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  8 23:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  8 23:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  8 23:59:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:00:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:00:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:00:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:00:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:00:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:00:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:00:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:00:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:00:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
